Information recording system and information recording method

ABSTRACT

In an information recording system, a sound processing unit generates a conversion candidate word in a process of converting sound information into text information. A recording unit records the text information and the conversion candidate word on a recording medium such that the text information and the conversion candidate word are associated with each other. A search unit performs a search based on a keyword and extracts a word matching the keyword from words within the text information and the conversion candidate word. A reading unit reads the text information including the word matching the keyword from the recording medium. A display unit displays the text information such that a part corresponding to the word matching the keyword and a part other than the corresponding part are able to be distinguished.

The present application is a continuation application based onInternational Patent Application No. PCT/JP2017/002706, filed on Jan.26, 2017, the content of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an information recording system and aninformation recording method.

Description of Related Art

In general, sounds in scenes such as conferences and interviews arerecorded and the recorded sounds are used thereafter. For example,minutes and articles can be created by reproducing sounds and creatingtext based on the sounds. With the development of sound recognitiontechnology, it has also become possible to generate text from a soundusing sound recognition software.

Conventional observation devices record only information of an objectobtained by observing the object. The record of a situation at anobservation site as well as the record of information obtained byobservation is regarded to be important to promote the use of data andprevent fraud. For example, there are examples of a laboratory notebookof a researcher, findings of a doctor, a construction site deploymentreport, and the like as examples of the record of the situation at theobservation site. Also, the declining birthrate and aging population andthe shortage of skilled workers have become a problem in every field.For the skill succession and education, the importance of recording ofan on-site situation is increasing more and more. It is possible torecord a situation of an observation site in more detail by recordingtext based on a sound of an observer in addition to information of anobject. Also, it is possible to efficiently extract the information ofthe object in a desired situation by searching text for a wordrepresenting the situation of the observation site.

On the other hand, technology for recording information of an object anda sound in association has been disclosed. For example, in thetechnology disclosed in Japanese Unexamined Patent Application, FirstPublication No. 2008-199079, the appearance of an object is imaged and asound uttered by an operator during imaging is acquired. The acquiredimage of the object and the acquired sound of the operator are recordedin association. In the technology disclosed in Japanese UnexaminedPatent Application, First Publication No. 2008-085582, an image and asound associated therewith are transmitted from a camera to a server.The server converts the received sound into text and generatesinformation to be added to the image on the basis of a conversionresult. The server stores the received image in association with theinformation generated on the basis of the sound.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention, an informationrecording system includes a sound acquisition unit, a sound processingunit, a recording unit, a keyword reception unit, a search unit, areading unit, and a display unit. The sound acquisition unit acquiressound information based on a sound uttered by a user. The soundprocessing unit converts the sound information acquired by the soundacquisition unit into text information and generates at least oneconversion candidate word corresponding to the sound in a process ofconverting the sound information into the text information. Therecording unit records the text information and the conversion candidateword such that the text information and the conversion candidate wordare associated with each other. The keyword reception unit receives akeyword. The search unit searches for the keyword received by thekeyword reception unit in the text information and the conversioncandidate word and extracts a word matching the keyword from wordswithin the text information and the conversion candidate word. Thereading unit reads the text information including a first partcorresponding to the word matching the keyword and a second part otherthan the first part from the recording medium. The display unit displaysthe text information read by the reading unit such that the first partand the second part are able to be distinguished.

According to a second aspect of the present invention, an informationrecording system includes an object information acquisition unit, asound acquisition unit, a sound processing unit, a recording unit, akeyword reception unit, a search unit, a reading unit, and a displayunit. The object information acquisition unit acquires objectinformation about an object. The sound acquisition unit acquires soundinformation based on a sound uttered by an observer who observes theobject. The sound processing unit converts the sound informationacquired by the sound acquisition unit into text information andgenerates at least one conversion candidate word corresponding to thesound in a process of converting the sound information into the textinformation. The recording unit records the object information, the textinformation, the conversion candidate word, and time point informationon a recording medium such that the object information, the textinformation, the conversion candidate word, and the time pointinformation are associated with each other. The time point informationindicates a time point at which the object information was acquired anda time point at which the sound information that is a source of the textinformation was acquired. The keyword reception unit receives a keyword.The search unit searches for the keyword received by the keywordreception unit in the text information and the conversion candidate wordand extracts a word matching the keyword from words within the textinformation and the conversion candidate word. The reading unit readsthe object information associated with the time point informationcorresponding to the word matching the keyword from the recordingmedium. The display unit displays the text information read by thereading unit.

According to a third aspect of the present invention, in the secondaspect, the recording unit may record the object information, the soundinformation, the text information, the conversion candidate word, andthe time point information on the recording medium such that the objectinformation, the sound information, the text information, the conversioncandidate word, and the time point information are associated with eachother. The time point information indicates a time point at which theobject information was acquired, a time point at which the soundinformation was acquired, and a time point at which the soundinformation that is a source of the text information was acquired. Theinformation recording system may further include an event detection unitconfigured to detect at least two events on the basis of at least onepiece of the object information and the sound information recorded onthe recording medium. The events are a state in which the at least onepiece of the object information and the sound information recorded onthe recording medium satisfies a predetermined condition. The readingunit may read at least two pieces of the object information, each ofwhich is associated with the time point information corresponding to anevent occurrence time point that is a time point at which the eventoccurred from the recording medium. The display unit may display theobject information read by the reading unit. The search unit may searchfor the keyword received by the keyword reception unit in the textinformation and the conversion candidate word that are associated withthe time point information corresponding to the event occurrence timepoint. The display unit may display the object information associatedwith the time point information corresponding to the event occurrencetime point at which the word matching the keyword was detected such thatthe object information is distinguished from other object information.

According to a fourth aspect of the present invention, in the secondaspect, the recording unit may record the object information, the soundinformation, the text information, the conversion candidate word, andthe time point information on the recording medium such that the objectinformation, the sound information, the text information, the conversioncandidate word, and the time point information are associated with eachother. The time point information indicates a time point at which theobject information was acquired, a time point at which the soundinformation was acquired, and a time point at which the soundinformation that is a source of the text information was acquired. Theinformation recording system may further include an event detection unitconfigured to detect at least two events on the basis of at least onepiece of the object information and the sound information recorded onthe recording medium. The events are a state in which the at least onepiece of the object information and the sound information recorded onthe recording medium satisfies a predetermined condition. The readingunit may read the object information and the text information that areassociated with the time point information corresponding to an eventoccurrence time point that is a time point at which the event occurredfrom the recording medium. The display unit may display the objectinformation and the text information read by the reading unit. Thesearch unit may search for the keyword received by the keyword receptionunit in the text information and the conversion candidate word that areassociated with the time point information corresponding to the eventoccurrence time point. The display unit may display the word that iswithin the text information and matches the keyword such that the wordis distinguished from other words within the text information, anddisplay the word that is within the text information and is associatedwith the conversion candidate word matching the keyword such that theword is distinguished from other words within the text information.

According to a fifth aspect of the present invention, in the thirdaspect, when a state of the object indicated by the object informationis a state predefined as an event detection condition, the eventdetection unit may detect the event.

According to a sixth aspect of the present invention, in the fourthaspect, when a state of the object indicated by the object informationis a state predefined as an event detection condition, the eventdetection unit may detect the event.

According to a seventh aspect of the present invention, in the thirdaspect, the sound information may be a time-series sound signal. Whenamplitude or power of the sound signal may exceed a threshold valuepredefined as an event detection condition, the event detection unit maydetect the event.

According to an eighth aspect of the present invention, in the fourthaspect, the sound information may be a time-series sound signal. Whenamplitude or power of the sound signal may exceed a threshold valuepredefined as an event detection condition, the event detection unit maydetect the event.

According to a ninth aspect of the present invention, in the thirdaspect, when a sound indicated by the sound information matches a soundof a keyword predefined as an event detection condition, the eventdetection unit may detect the event.

According to a tenth aspect of the present invention, in the fourthaspect, when a sound indicated by the sound information matches a soundof a keyword predefined as an event detection condition, the eventdetection unit may detect the event.

According to an eleventh aspect of the present invention, in the secondaspect, the reading unit may read the object information and the textinformation that are associated with the time point informationcorresponding to the word matching the keyword from the recordingmedium. The display unit may display the object information and the textinformation read by the reading unit. The display unit may display theword that is within the text information and matches the keyword suchthat the word is distinguished from other words within the textinformation and display the word that is within the text information andis associated with the conversion candidate word matching the keywordsuch that the word is distinguished from other words within the textinformation.

According to a twelfth aspect of the present invention, in the secondaspect, the recording unit may record the object information, the soundinformation, and the time point information on the recording medium suchthat the object information, the sound information, and the time pointinformation are associated with each other. The time point informationindicates a time point at which the object information was acquired anda time point at which the sound information was acquired. The readingunit may read the sound information from the recording medium. The soundprocessing unit may convert the sound information read by the readingunit into the text information and generates the conversion candidateword. The recording unit may associate the text information and theconversion candidate word with the object information and the time pointinformation recorded on the recording medium and record the textinformation and the conversion candidate word on the recording medium.The time point information with which the text information is associatedindicates a time point at which the sound information that is a sourceof the text information was acquired.

According to a thirteenth aspect of the present invention, aninformation recording method includes a sound acquisition step, a soundprocessing step, a recording step, a keyword reception step, a searchstep, a reading step, and a display step. In the sound acquisition step,a sound acquisition unit acquires sound information based on a sounduttered by a user. In the sound processing step, a sound processing unitconverts the sound information acquired by the sound acquisition unitinto text information and generates at least one conversion candidateword corresponding to the sound in a process of converting the soundinformation into the text information. In the recording step, arecording unit records the text information and the conversion candidateword on a recording medium such that the text information and theconversion candidate word are associated with each other. In the keywordreception step, a keyword reception unit receives a keyword. In thesearch step, a search unit searches for the keyword received by thekeyword reception unit in the text information and the conversioncandidate word and extracts a word matching the keyword from wordswithin the text information and the conversion candidate word. In thereading step, a reading unit reads the text information including afirst part corresponding to the word matching the keyword and a secondpart other than the first part from the recording medium. In the displaystep, a display unit displays the text information read by the readingunit such that the first part and the second part are able to bedistinguished.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of an informationrecording system according to a first embodiment of the presentinvention.

FIG. 2 is a flowchart showing a procedure of processing of theinformation recording system according to the first embodiment of thepresent invention.

FIG. 3 is a diagram showing a schematic configuration of a microscopesystem according to the first embodiment of the present invention.

FIG. 4 is a reference diagram showing a search method of a search unitin the information recording system according to the first embodiment ofthe present invention.

FIG. 5 is a reference diagram showing a screen of a display unit in theinformation recording system according to the first embodiment of thepresent invention.

FIG. 6 is a reference diagram showing a relationship between an eventoccurrence time point and an event period in the information recordingsystem according to the first embodiment of the present invention.

FIG. 7 is a block diagram showing a configuration of an informationrecording system according to a first modified example of the firstembodiment of the present invention.

FIG. 8 is a block diagram showing a configuration of an informationrecording system according to a second modified example of the firstembodiment of the present invention.

FIG. 9 is a block diagram showing a configuration of an informationrecording system according to a third modified example of the firstembodiment of the present invention.

FIG. 10 is a flowchart showing a procedure of processing of theinformation recording system according to the third modified example ofthe first embodiment of the present invention.

FIG. 11 is a reference diagram showing event detection based on objectinformation in the information recording system according to the thirdmodified example of the first embodiment of the present invention.

FIG. 12 is a reference diagram showing event detection based on objectinformation in the information recording system according to the thirdmodified example of the first embodiment of the present invention.

FIG. 13 is a reference diagram showing event detection based on soundinformation in the information recording system according to the thirdmodified example of the first embodiment of the present invention.

FIG. 14 is a reference diagram showing event detection based on soundinformation in the information recording system according to the thirdmodified example of the first embodiment of the present invention.

FIG. 15 is a reference diagram showing a screen of a display unit in theinformation recording system according to the third modified example ofthe first embodiment of the present invention.

FIG. 16 is a block diagram showing a configuration of an informationrecording system according to a second embodiment of the presentinvention.

FIG. 17 is a block diagram showing a configuration of an informationrecording device according to the second embodiment of the presentinvention.

FIG. 18 is a flowchart showing a procedure of processing of theinformation recording device according to the second embodiment of thepresent invention.

FIG. 19 is a block diagram showing a configuration of an informationrecording system according to a third embodiment of the presentinvention.

FIG. 20 is a flowchart showing a procedure of processing of theinformation recording system according to the third embodiment of thepresent invention.

FIG. 21 is a reference diagram showing generation of determined wordsand conversion candidate words by a sound processing unit and asituation of a search by a search unit in the information recordingsystem according to the third embodiment of the present invention.

FIG. 22 is a reference diagram showing text information displayed by adisplay unit in the information recording system according to the thirdembodiment of the present invention.

FIG. 23 is a block diagram showing a configuration of an informationrecording system according to a fourth embodiment of the presentinvention.

FIG. 24 is a flowchart showing a procedure of processing of aninformation recording device according to the fourth embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention will be described with reference tothe drawings.

First Embodiment

FIG. 1 shows a configuration of an information recording system 10according to a first embodiment of the present invention. As shown inFIG. 1, the information recording system 10 includes an objectinformation acquisition unit 20, an image acquisition unit 30, a soundacquisition unit 40, a sound processing unit 50, a keyword receptionunit 55, a recording unit 60, a search unit 65, a recording medium 70, areading unit 80, a display unit 90, and a sound output unit 100.

The object information acquisition unit 20 acquires object informationabout an object. The object is an object to be observed. The observationis an act of figuring out a state of the object. The observation mayinclude acts such as diagnosis, an examination, and an inspection. Theobject information acquired for observation may not be necessarilyvisual information of the outside or inside of the object, i.e., imageinformation. For example, the object information acquisition unit 20 isa camera mounted on image devices such as a microscope, an endoscope, athermal imaging device, an X-ray device, and a computed tomography (CT)device. These image devices acquire image information of the object.These image devices may include a camera that generates imageinformation on the basis of a signal obtained from a sensor. The imageinformation acquired by these image devices may be any one ofmoving-image information and still-image information. The objectinformation acquisition unit 20 may be a sensor that acquiresinformation such as a temperature, acceleration, pressure, a voltage,and a current of the object. When the object is a living thing, theobject information acquisition unit 20 may be a vital sensor thatacquires vital information of the object. For example, the vitalinformation is information such as a body temperature, blood pressure, apulse, an electrocardiogram, or a degree of blood oxygen saturation. Theobject information acquisition unit 20 may be a microphone that acquiressound information based on a sound uttered by the object. For example,the sound information is information of a hammering test sound, an echosound, a heart sound, noise, and the like. Additional information suchas time point information may be added to the object informationacquired by the object information acquisition unit 20. For example, theobject information acquisition unit 20 adds time point informationindicating a time point at which the object information was acquired tothe object information, and outputs the object information to which thetime point information is added. When the object information istime-series information, time point information for identifying aplurality of different time points is added to the object information.For example, the time point information associated with the objectinformation includes a time point at which acquisition of the objectinformation was started and a sampling rate.

The image acquisition unit 30 acquires image information indicating atype of situation in which the object information was acquired. Theimage information acquired by the image acquisition unit 30 indicates astate of at least one of the object and surroundings of the object whenthe object information is acquired. That is, the image informationacquired by the image acquisition unit 30 indicates an observationsituation. The image acquisition unit 30 is an image device including acamera. The image acquisition unit 30 acquires the image information inparallel with the acquisition of the object information by the objectinformation acquisition unit 20. The image information acquired by theimage acquisition unit 30 may be any one of moving image-information andstill-image information. For example, the image acquisition unit 30acquires the image information including an image of the at least one ofthe object and the surroundings of the object. For example, thesurroundings of the object include a device on which the objectinformation acquisition unit 20 is mounted. In this case, imageinformation including an image of at least one of the object and thedevice on which the object information acquisition unit 20 is mounted isacquired. The surroundings of the object may also include an observerwho observes the object. In this case, image information including animage of at least one of the object and the observer is acquired. Theimage acquisition unit 30 is disposed such that at least one of theobject and the surroundings of the object is included in a photographingrange.

When the image information acquired by the image acquisition unit 30includes an image of the object, the image information includes an imageof a part or all of the object. When the image information acquired bythe image acquisition unit 30 includes an image of the device on whichthe object information acquisition unit 20 is mounted, the imageinformation includes an image of a part or all of the device. When theimage information acquired by the image acquisition unit 30 includes animage of a user, the image information includes an image of a part orall of the user. When the object information acquisition unit 20 is animage device and the object information is an image of the object, aphotographic visual field of the image acquisition unit 30 is wider thanthat of the object information acquisition unit 20. For example, theobject information acquisition unit 20 acquires image information of apart of the object and the image acquisition unit 30 acquires imageinformation of all of the object. The image acquisition unit 30 may be awearable camera worn by the user, i.e., an observer. For example, thewearable camera is a head mount type camera mounted in the vicinity ofthe eyes of the observer such that image information corresponding tothe viewpoint of the observer can be acquired. Therefore, the imageacquisition unit 30 may be disposed at a position of the viewpoint ofthe observer who observes the object or in the vicinity of theviewpoint. Additional information such as time point information may beadded to the image information acquired by the image acquisition unit30. For example, the image acquisition unit 30 adds the time pointinformation indicating a time point at which the image information wasacquired to the image information and outputs the image information towhich the time point information is added. When the image information istime-series information, time point information for identifying aplurality of different time points is added to the image information.For example, the time point information associated with the imageinformation includes a time point at which acquisition of the imageinformation was started and a sampling rate.

The sound acquisition unit 40 acquires sound information based on asound uttered by the observer who observes the object. For example, thesound acquisition unit 40 is a microphone. The sound acquisition unit 40may be a wearable microphone worn by the observer. The wearablemicrophone is worn in the vicinity of the observer's mouth. The soundacquisition unit 40 may be a microphone having directivity such thatonly the sound of the observer is acquired. In this case, the soundacquisition unit 40 may not be installed in the vicinity of theobserver's mouth. Thereby, a degree of freedom with respect to thedisposition of the sound acquisition unit 40 is obtained. Because noiseother than the sound of the observer is eliminated, the efficiency ingeneration and search of text information is improved. In parallel withthe acquisition of the object information by the object informationacquisition unit 20, the sound acquisition unit 40 acquires soundinformation. Additional information such as time point information maybe added to the sound information acquired by the sound acquisition unit40. For example, the sound acquisition unit 40 adds the time pointinformation indicating a time point at which the sound information wasacquired to the sound information and outputs the sound information towhich the time point information is added. When the sound information istime-series information, the time point information for identifying aplurality of different time points is added to the sound information.For example, the time point information associated with the soundinformation includes a time point at which acquisition of the soundinformation was started and a sampling rate.

The sound processing unit 50 converts the sound information acquired bythe sound acquisition unit 40 into text information and generates atleast one conversion candidate word corresponding to the sound in aprocess of converting the sound information into the text information.For example, the sound processing unit 50 includes a sound processingcircuit that performs sound processing. The sound processing unit 50includes a sound recognition unit 500 and a text generation unit 510.The sound recognition unit 500 recognizes the sound of the user, i.e.,the observer, on the basis of the sound information acquired by thesound acquisition unit 40. The text generation unit 510 generates textinformation corresponding to the user's sound by converting the soundrecognized by the sound recognition unit 500 into the text information.The text generation unit 510 may divide consecutive sounds intoappropriate blocks and generate text information for each block. Forexample, a block constitutes one sentence or clause. The block mayinclude a part satisfying a predetermined condition. For example, theblock may include an utterance part corresponding to a sound “ah”, asilence part, or the like. The block may correspond to an event to bedescribed below. Also, the text generation unit 510 generates at leastone conversion candidate word corresponding to the sound in the processof converting the sound information into the text information. Forexample, the text generation unit 510 extracts candidate wordsrecognized by the sound recognition unit 500 with predetermined accuracyor more. The text generation unit 510 selects a word with the highestaccuracy among the candidate words as a determined word. The textgeneration unit 510 generates text information including the determinedword. Also, the text generation unit 510 generates a word other than thedetermined word among the candidate words as a conversion candidateword. Pronunciation of a conversion candidate word is similar topronunciation of a determined word. The generated conversion candidateword is associated with the determined word constituting the textinformation. The sound processing unit 50 (the text generation unit 510)outputs the text information and the conversion candidate word.Additional information such as time point information may be added tothe text information generated by the sound processing unit 50. Forexample, the sound processing unit 50 (the text generation unit 510)adds time point information indicating a time point at which the textinformation was generated to the text information and outputs the textinformation to which the time point information is added. At this time,the sound processing unit 50 (the text generation unit 510) adds thetime point information to the determined word. Alternatively, the soundprocessing unit 50 (the text generation unit 510) adds the time pointinformation to the conversion candidate word and the determined word.That is, the sound processing unit 50 (the text generation unit 510)adds the time point information to the text information for each word.The time point information may be added to the text information for eachblock or for each event to be described below. When the text informationis time-series information, time point information corresponding to aplurality of different time points is added to the text information. Thetime point of the text information corresponds to a start time point ofthe sound information associated with the text information.

The object information acquired by the object information acquisitionunit 20, the image information acquired by the image acquisition unit30, the sound information acquired by the sound acquisition unit 40, thetext information generated by the sound processing unit 50, and theconversion candidate word generated by the sound processing unit 50 areinput to the recording unit 60. The recording unit 60 records the objectinformation, the image information, the sound information, the textinformation, the conversion candidate word, and the time pointinformation on the recording medium 70 such that the object information,the image information, the sound information, the text information, theconversion candidate word, and the time point information are associatedwith each other. At this time, the recording unit 60 associates theobject information, the image information, the sound information, thetext information, and the conversion candidate word with each other onthe basis of the time point information. For example, the recording unit60 includes a recording processing circuit that performs an informationrecording process. At least one piece of the object information, theimage information, the sound information, the text information, and theconversion candidate word may be compressed. Therefore, the recordingunit 60 may include a compression processing circuit for compressinginformation. The recording unit 60 may include a buffer for therecording process and the compression process. The time pointinformation indicates time points at which the object information, theimage information, and the sound information were acquired. The timepoint information associated with the text information indicates thetime point at which the sound information that is a source of the textinformation was acquired. For example, the time point information isadded to each piece of the object information, the image information,the sound information, and the text information. The object information,the image information, the sound information, and the text informationare associated with each other via the time point information. Thedetermined word and the conversion candidate word that constitute thetext information are associated with each other. The time pointinformation may be added to the conversion candidate word. In this case,the object information, the image information, the sound information,the text information, and the conversion candidate word are associatedwith each other via the time point information.

The object information, the image information, the sound information,the text information, and the conversion candidate word are associatedwith each other as information about a common object. The objectinformation, the image information, the sound information, the textinformation, and the conversion candidate word may be associated witheach other as information about a plurality of objects related to eachother. For example, each piece of the object information, the imageinformation, the sound information, the text information, and theconversion candidate word includes one file and the recording unit 60records each file on the recording medium 70. In this case, informationfor associating files of the object information, the image information,the sound information, the text information, and the conversioncandidate word is recorded on the recording medium 70.

The recording medium 70 is a nonvolatile storage device. For example,the recording medium 70 is at least one of an erasable programmable readonly memory (EPROM), an electrically erasable programmable read-onlymemory (EEPROM), a flash memory, and a hard disk drive. The recordingmedium 70 may not be disposed at an observation site. For example, theinformation recording system 10 may have a network interface and theinformation recording system 10 may be connected to the recording medium70 via a network such as the Internet or a local area network (LAN). Theinformation recording system 10 may have a wireless communicationinterface and the information recording system 10 may be connected tothe recording medium 70 through wireless communication according to astandard such as Wi-Fi (registered trademark) or Bluetooth (registeredtrademark). Therefore, the information recording system 10 may notdirectly include the recording medium 70.

The keyword reception unit 55 receives a keyword. For example, thekeyword reception unit 55 is configured as an operation unit. Forexample, the operation unit is configured to include at least one of abutton, a switch, a key, a mouse, a joystick, a touch pad, a track ball,and a touch panel. The keyword reception unit 55 may be configured as acommunication unit that wirelessly communicates with the operation unit.The user inputs a keyword for a search via the keyword reception unit55. A keyword includes one or more words.

The search unit 65 searches for a keyword received by the keywordreception unit 55 in the text information and the conversion candidateword and extracts a word matching the keyword from the word (thedetermined word) within the text information and the conversioncandidate word. For example, the search unit 65 includes an informationprocessing circuit that performs information processing. For example,the text information and the conversion candidate word recorded on therecording medium 70 are read by the reading unit 80. The search unit 65searches for the keyword in the text information and the conversioncandidate word read by the reading unit 80. As a result of the search,any one of the determined word and the conversion candidate word canmatch the keyword. Alternatively, neither the determined word nor theconversion candidate word may match the keyword. Also, the time pointinformation recorded on the recording medium 70 is read by the readingunit 80. When the determined word matches the keyword, the search unit65 identifies the time point information associated with the determinedword. Thereby, the search unit 65 identifies an utterance time pointthat is a time point at which a sound corresponding to the keyword wasuttered. When the conversion candidate word matches the keyword, thesearch unit 65 identifies the conversion candidate word or the timepoint information associated with the determined word correspondingthereto. When the time point information is associated with textinformation for each block, the search unit 65 identifies the time pointinformation associated with the block. When the time point informationis associated with the text information for each event to be describedbelow, the search unit 65 identifies the time point informationassociated with the event. The recording unit 60 may record theutterance time point identified by the search unit 65 on the recordingmedium 70.

The reading unit 80 reads the object information, the image information,the sound information, and the text information from the recordingmedium 70. Thereby, the reading unit 80 reproduces the objectinformation, the image information, the sound information, and the textinformation recorded on the recording medium 70. For example, thereading unit 80 includes a reading processing circuit that performs aninformation reading process. At least one piece of the objectinformation, the image information, the sound information, and the textinformation recorded on the recording medium 70 may be compressed.Therefore, the reading unit 80 may include a decompression processingcircuit for decompressing the compressed information. The reading unit80 may include a buffer for a reading process and a decompressionprocess. The reading unit 80 reads the object information, the imageinformation, the sound information, and the text information associatedwith the time point information corresponding to a word matching akeyword. For example, the reading unit 80 reads the object information,the image information, the sound information, and the text informationassociated with the same time point information corresponding to theutterance time point identified by the search unit 65, from therecording medium 70. When pieces of time point information associatedwith the information are not synchronized with each other, the readingunit 80 may read the information in consideration of a difference in thetime point information with respect to a reference time point.

The display unit 90 displays the object information, the imageinformation, and the text information read by the reading unit 80 suchthat the object information, the image information, and the textinformation are associated with each other. The display unit 90 is adisplay device such as a liquid crystal display. For example, thedisplay unit 90 is a monitor of a personal computer (PC). The displayunit 90 may be a wearable display such as smart glasses worn by theuser. The display unit 90 may be a display unit of a device on which theobject information acquisition unit 20 is mounted. The display unit 90may be a large-size monitor for sharing information. The display unit 90may be a touch panel display. For example, the display unit 90simultaneously displays the object information, the image information,and the text information. At this time, the display unit 90 displays theobject information, the image information, and the text information in astate in which these pieces of information are arranged. Informationselected from a series of object information, image information, andtext information corresponding to the same block or the event to bedescribed below and associated with each other may be arranged anddisplayed on the display unit 90. Also, the user may be able to switchthe information displayed on the display unit 90. For example, theobject information acquired by the sensor or the vital sensor includestime-series sensor signals. For example, the display unit 90 displays awaveform of a sensor signal as a graph.

The display unit 90 may visually display the sound information read bythe reading unit 80 as character information or a chart. In this case,the display unit 90 displays the object information, the imageinformation, the sound information, and the text information read by thereading unit 80 such that the object information, the image information,the sound information, and the text information are associated with eachother. For example, the sound information includes time-series soundsignals. For example, the display unit 90 displays a change in amplitudeor power of the sound signal over time as a graph.

The sound output unit 100 outputs a sound based on the sound informationread by the reading unit 80. For example, the sound output unit 100 is aspeaker.

When the object information acquired by the object informationacquisition unit 20 is image information, the object information may beoutput to the display unit 90. The display unit 90 may display theobject information in parallel with the acquisition of the objectinformation by the object information acquisition unit 20. The imageinformation acquired by the image acquisition unit 30 may be output tothe display unit 90. The display unit 90 may display the imageinformation acquired by the image acquisition unit 30 in parallel withthe acquisition of the object information by the object informationacquisition unit 20. Thereby, the user can figure out a state of theobject and an observation situation in real time.

The sound processing unit 50, the recording unit 60, the search unit 65,and the reading unit 80 may include one or more processors. For example,the processor is at least one of a central processing unit (CPU), adigital signal processor (DSP), and a graphics processing unit (GPU).The sound processing unit 50, the recording unit 60, the search unit 65,and the reading unit 80 may include an application specific integratedcircuit (ASIC) or a field-programmable gate array (FPGA).

In the information recording system 10, the image informationacquisition and recording are optional. Therefore, the informationrecording system 10 may not include the image acquisition unit 30. Inthis case, the recording unit 60 records the object information, thesound information, the text information, the conversion candidate word,and the time point information on the recording medium 70 such that theobject information, the sound information, the text information, theconversion candidate word, and the time point information are associatedwith each other. The time point information indicates time points atwhich the object information and the sound information were acquired andalso indicates a time point at which the sound information that is asource of the text information was acquired. The reading unit 80 readsthe object information, the sound information, and the text informationassociated with the time point information corresponding to the wordmatching the keyword from the recording medium 70. The display unit 90displays the object information and the text information read by thereading unit 80 such that the object information and the textinformation are associated with each other. The sound output unit 100outputs a sound based on the sound information read by the reading unit80.

The information recording system 10 may not include the sound outputunit 100 and the recording unit 60 may not record the sound information.In this case, the recording unit 60 records the object information, theimage information, the text information, the conversion candidate word,and the time point information on the recording medium 70 such that theobject information, the image information, the text information, theconversion candidate word, and the time point information are associatedwith each other. The time point information indicates time points atwhich the object information and the image information were acquired andalso indicates a time point at which the sound information that is asource of the text information was acquired. The reading unit 80 readsthe object information, the image information, and the text informationassociated with the time point information corresponding to the wordmatching the keyword from the recording medium 70. The display unit 90displays the object information, the image information, and the textinformation read by the reading unit 80 such that the objectinformation, the image information, and the text information areassociated with each other.

FIG. 2 shows a procedure of processing of the information recordingsystem 10. The procedure of processing of the information recordingsystem 10 will be described with reference to FIG. 2.

The object information acquisition unit 20 acquires object informationabout an object (step S100 (an object information acquisition step)).The object information acquired in step S100 is stored in the bufferwithin the recording unit 60. In parallel with the acquisition of theobject information by the object information acquisition unit 20, theimage acquisition unit 30 acquires image information indicating a typeof situation in which the object information was acquired (step S105 (animage acquisition step)). The image information acquired in step S105 isstored in the buffer within the recording unit 60. In parallel with theacquisition of the object information by the object informationacquisition unit 20, the processing in step S110 is performed. Step S110includes step S111 (a sound acquisition step) and step S112 (a soundprocessing step). In step S111, the sound acquisition unit 40 acquiressound information based on a sound uttered by the observer who observesthe object. In step S112, the sound processing unit 50 converts thesound information acquired by the sound acquisition unit 40 into textinformation and generates at least one conversion candidate wordcorresponding to a sound in a process of converting the soundinformation into the text information. In step S110, the processing instep S111 and step S112 is iterated. The sound information acquired instep S111, the text information generated in step S112, and theconversion candidate word generated in step S112 are stored in thebuffer within the recording unit 60. Also, the time point informationcorresponding to the time point at which the information was generatedis stored in the buffer within the recording unit 60.

Processing start timings of step S100, step S105, and step S110 may notbe the same. Processing end timings of step S100, step S105, and stepS110 may not be the same. At least some of periods during which theprocessing in step S100, step S105, and step S110 is performed overlapeach other.

After the acquisition of the object information, the image information,and the sound information is completed, the recording unit 60 recordsthe object information, the image information, the sound information,the text information, the conversion candidate word, and the time pointinformation stored in the buffer within the recording unit 60 on therecording medium 70 such that the object information, the imageinformation, the sound information, the text information, the conversioncandidate word, and the time point information are associated with eachother (step S115 (a recording step)).

After step S115, the keyword reception unit 55 receives a keyword (stepS120 (a keyword reception step)).

After step S120, the search unit 65 searches for the keyword received bythe keyword reception unit 55 in the text information and the conversioncandidate word, and extracts a word matching the keyword from the wordswithin the text information and the conversion candidate word (step S125(a search step)).

After step S125, the reading unit 80 reads the object information, theimage information, the sound information, and the text informationassociated with the time point information corresponding to the wordmatching the keyword from the recording medium 70 (step S130 (a readingstep)). The user may be able to specify a timing at which theinformation is read.

After step S130, the display unit 90 displays the object information,the image information, and the text information read by the reading unit80 such that the object information, the image information, and the textinformation are associated with each other. Also, the sound output unit100 outputs a sound based on the sound information read by the readingunit 80 (step S135 (a display step and a sound output step)).

When the information recording system 10 does not include the imageacquisition unit 30, the processing in step S105 is not performed. Also,in step S115, the recording unit 60 records the object information, thesound information, the text information, the conversion candidate word,and the time point information on the recording medium 70 such that theobject information, the sound information, the text information, theconversion candidate word, and the time point information are associatedwith each other. In step S130, the reading unit 80 reads the objectinformation, the sound information, and the text information associatedwith the time point information corresponding to the word matching thekeyword from the recording medium 70. In step S135, the display unit 90displays the object information and the text information read by thereading unit 80 in step S130 such that the object information and thetext information are associated with each other. Also, in step S135, thesound output unit 100 outputs a sound based on the sound informationread by the reading unit 80 in step S130.

When the information recording system 10 does not include the soundoutput unit 100 and the recording unit 60 does not record the soundinformation, the recording unit 60 records the object information, theimage information, the text information, the conversion candidate word,and the time point information on the recording medium 70 such that theobject information, the image information, the text information, theconversion candidate word, and the time point information are associatedwith each other in step S115. In step S130, the reading unit 80 readsthe object information, the image information, and the text informationassociated with the time point information corresponding to the wordmatching the keyword from the recording medium 70. In step S135, thedisplay unit 90 displays the object information, the image information,and the text information read by the reading unit 80 in step S130 suchthat the object information, the image information, and the textinformation are associated with each other.

At a conventional observation site, the user records a situation of anobservation site by writing by hand. On the other hand, because usersare working in various environments and situations at a site, there is acase in which it is difficult to record at a site. There are cases inwhich the users cannot use their hands for reasons of safety or hygiene.In these cases, there is a possibility that a user will cause omissionsin recording or erroneous recording due to recording an on-sitesituation on the basis of ambiguous memory after observation.

As described above, the object information is acquired by the objectinformation acquisition unit 20 and the image acquisition unit 30acquires the image information indicating a type of situation in whichthe object information was acquired. The acquired object information andimage information are recorded on the recording medium 70 by therecording unit 60. Thereby, the information recording system 10 canrecord visual information indicating a type of situation in which theobject information was acquired.

In the above-described method, a burden on the user for recording theinformation indicating a type of situation in which the objectinformation was acquired is small. Even when the user cannot use his/herhand, necessary information can be recorded and omissions in recordingor erroneous recording are reduced. Therefore, the information recordingsystem 10 can accurately and efficiently leave a record showing the typeof situation in which the object information was acquired.

In the above-described method, the user's comments when the objectinformation was acquired are recorded as a sound and text correspondingto the sound is recorded in association with the object information andthe image information. A “tag” based on the text is attached to theobject information, the image information, and the sound information andtherefore the user can easily understand a situation when theinformation was acquired.

In the above-described method, the conversion candidate word generatedin the process of converting the sound information into the textinformation is recorded on the recording medium 70 by the recording unit60. The search unit 65 searches for the keyword received by the keywordreception unit 55 in the text information and the conversion candidateword and extracts a word matching the keyword from the words within thetext information and the conversion candidate word. Even when a worderroneously recognized in a sound recognition process cannot beextracted from the text information through a search, the search unit 65can extract the conversion candidate word corresponding to theerroneously recognized word. Therefore, the information recording system10 can reduce omissions in a word search in text information. As aresult, the information recording system 10 can provide the user withobject information, image information, sound information, and textinformation corresponding to the keyword received by the keywordreception unit 55.

A specific example of the information recording system 10 will bedescribed below. FIG. 3 shows a schematic configuration of a microscopesystem 11 which is an example of the information recording system 10. Asshown in FIG. 3, the microscope system 11 includes a microscope 200, acamera 31 a, a camera 31 b, a camera 31 c, a microphone 41, a server201, and a PC 202.

The microscope 200 is a device for enlarging and observing an objectOB1. The camera 21 connected to the microscope 200 constitutes theobject information acquisition unit 20. The camera 21 acquires imageinformation of the object OB1 enlarged by the microscope 200 as objectinformation. For example, the camera 21 acquires moving-imageinformation.

The camera 31 a, the camera 31 b, and the camera 31 c constitute theimage acquisition unit 30. A photographic visual field of each of thecamera 31 a, the camera 31 b, and the camera 31 c is wider than that ofthe camera connected to the microscope 200. For example, the camera 31a, the camera 31 b, and the camera 31 c acquire moving-imageinformation.

The camera 31 a is disposed in the vicinity of a tip of an objectivelens of the microscope 200. The camera 31 a acquires image informationincluding an image of the object OB1 and the tip of the objective lensof the microscope 200 by photographing the vicinity of the tip of theobjective lens of the microscope 200. Thereby, a positional relationshipbetween the object OB1 and the tip of the objective lens of themicroscope 200 is recorded as image information. The user who is theobserver does not need to approach the object OB1 and the tip of theobjective lens of the microscope 200 to check states thereof. By viewingthe image information acquired by the camera 31 a, the user can easilyfigure out a situation such as which part of the object OB1 is beingobserved or how close the objective lens tip of the microscope 200 is tothe object OB1.

The camera 31 b is disposed in an indoor space where observation isperformed. The camera 31 b acquires image information including an imageof all of the object OB1 and the microscope 200 by photographing all ofthe object OB1 and the microscope 200. Thereby, all situations of anobservation site are recorded as the image information. By viewing theimage information acquired by the camera 31 b, the user can easilyfigure out a situation such as an event occurring in a portion differentfrom a portion to which the user is paying attention. When the objectOB1 is a living thing, the state of the object OB1 is likely to affectthe object information obtained by observation. For example, even whenit is difficult to determine a state related to death and life of theobject OB1 from the object information, the user can easily figure outthe state of the object OB1 by viewing the image information acquired bythe camera 31 b. The camera 31 b may acquire image information includingan image of the user.

The camera 31 c is configured as a wearable camera. The camera 31 c isconfigured as the wearable camera by being attached to an accessory 203capable of being attached to the user's head. When the user wears theaccessory 203, the camera 31 c is disposed at a position near aviewpoint of the user. The camera 31 c acquires image informationincluding an image of the object OB1 and the microscope 200 byphotographing the object OB1 and the microscope 200. Alternatively, thecamera 31 c acquires image information including an image of themicroscope 200 without including an image of the object OB1 byphotographing the microscope 200. Thereby, an observation situationcorresponding to a part to which the user is paying attention inobservation is recorded as the image information. Thereby, themicroscope system 11 can record observation states such as a situationbefore the object OB1 is set up on a microscope stage, a procedure ofadjusting the microscope 200, and an adjustment state of the microscope200. The user, other people, and the like can easily figure out asituation during the observation in real time or after the end ofobservation by viewing the recorded observation states.

The microphone 41 constitutes the sound acquisition unit 40. Themicrophone 41 is configured as a wearable microphone by being attachedto the accessory 203.

The server 201 includes the sound processing unit 50, the keywordreception unit 55, the recording unit 60, the search unit 65, therecording medium 70, and the reading unit 80. The object informationacquired by the camera 21, the image information acquired by the camera31 a, the camera 31 b, and the camera 31 c, and the sound informationacquired by the microphone 41 are input to the server 201.

The PC 202 is connected to the server 201. The screen 91 of the PC 202constitutes the display unit 90. The smart glasses may constitute thedisplay unit 90. In parallel with the acquisition of the objectinformation, the smart glasses may display the image information that isthe object information and the image information acquired by each of thecamera 31 a, the camera 31 b, and the camera 31 c. By wearing the smartglasses, the user can figure out the state of the object OB1 and theobservation situation in real time.

The information recording system 10 may be applied to a microscopesystem using a multiphoton excitation fluorescence microscope. Themultiphoton excitation fluorescence microscope is used within a darkroom. A camera connected to the multiphoton excitation fluorescencemicroscope constitutes the object information acquisition unit 20. Forexample, as infrared cameras, the camera 31 a, the camera 31 b, and thecamera 31 c constitute the image acquisition unit 30. The infraredcamera acquires image information including an image of all of theobject and the multiphoton excitation fluorescence microscope byphotographing all of the object and the multiphoton excitationfluorescence microscope. For example, the user who is an observer wearsa wearable microphone constituting the sound acquisition unit 40. Adevice such as a PC includes the sound processing unit 50, the keywordreception unit 55, the recording unit 60, the search unit 65, therecording medium 70, and the reading unit 80. The object informationacquired by the camera connected to the multiphoton excitationfluorescence microscope, the image information acquired by the infraredcamera, and the sound information acquired by the wearable microphoneare input to the device. The screen of the device constitutes thedisplay unit 90.

In a dark environment, it is difficult for the user to figure out thestate of the microscope and the situation of the experiment and writethe state and the situation that have been figured out on paper with theuser's hand. In a system to which the information recording system 10 isapplied, the user does not need to stop the experiment and turn on alight in order to know the state of the microscope and the situation ofthe experiment. Also, the user does not need to temporarily stop themicroscope and look into the dark room. Also, the user does not need tomanually write the state of the microscope and the situation of theexperiment on paper with his/her hand.

The information recording system 10 may be applied to an endoscopesystem. An endoscope is a device for observing the inside of a body of aperson, i.e., a patient. The user, i.e., the doctor, utters commentssimultaneously with an inspection by the endoscope. The comments utteredby the user can be used to create accurate inspection records. Theseinspection records are used for the purpose of creating findings,materials for conference presentation, educational content for lessexperienced doctors, or the like.

The information recording system 10 may be applied to a medicalexamination system at a critical care emergency site. In the examinationsystem, a vital sensor is worn on the person to be examined, i.e., thepatient. The vital sensor acquires biological information such as a bodytemperature, blood pressure, and a pulse of the patient as objectinformation. The user, i.e., the doctor, utters comments simultaneouslywith the acquisition of the object information by the vital sensor. Thecomments uttered by the user can be used to accurately and efficientlydeliver findings at a site with respect to the patient to other peoplesuch as other doctors.

The information recording system 10 may be applied to an inspectionsystem. In the inspection system, the probe acquires a signal such as acurrent according to a defect on the surface of an industrial productsuch as an aircraft fuselage as the object information. The user utterscomments simultaneously with the acquisition of the object informationby the probe. The comments uttered by the user can be used to create awork report with respect to examination. The information recordingsystem 10 may be applied to an inspection system using an industrialendoscope. The industrial endoscope acquires image information ofobjects such as scratches and corrosion inside hollow objects such asboilers, turbines, engines, and chemical plants.

The information recording system 10 may be applied to a work recordingsystem. In the work recording system, a camera acquires imageinformation of an object such as a circuit board as the objectinformation. The user utters comments simultaneously with theacquisition of the object information by the camera. The commentsuttered by the user can be used to create accurate work records. Thework records are used for the purpose of creating a work report relatedto work with respect to the object and educational content for lessexperienced workers. Also, the user can easily trace work on the basisof a work history when a problem or the like occurs by storing the workrecord as the work history.

Hereinafter, a search method of the search unit 65 will be described.FIG. 4 shows the search method of the search unit 65. In the exampleshown in FIG. 4, a keyword “experiment condition” is specified. Thekeyword “experiment condition” is pronounced as “jikken jyouken” inJapanese. In the example shown in FIG. 4, the keyword “experimentcondition” is recognized as the term “ten requirements” without beingcorrectly recognized in the sound recognition process. The term “tenrequirements” is pronounced as “jyukken youken” in Japanese. The term“ten requirements” is included in the text information. The keyword“experiment condition” is not included in the text information. Thus,when a search has been performed under the keyword “experimentcondition” according to the conventional technology, a part includingthe term “ten requirements” that is erroneously recognized is notextracted from the text information. That is, omissions in a searchoccur.

The keyword “experiment condition” is divided into two words“experiment” and “condition”. The word “experiment” is pronounced as“jikken” in Japanese. The word “condition” is pronounced as “jyouken” inJapanese. Three conversion candidate words “experiment”, “case”, and“examination” are associated with the word “ten” in the text informationwhich is the determined word after the sound recognition process. Theseare examples of words that are similar in pronunciation and easilymisrecognized. The word “case” is pronounced as “jiken” in Japanese. Theword “examination” is pronounced as “jyuken” in Japanese. The word“experiment” included in the keyword matches the word “experiment” inthe conversion candidate word. Likewise, three conversion candidatewords “securities”, “condition”, and “experiment” whose pronunciationsare similar to each other are associated with the word “requirements” inthe text information. The word “securities” is pronounced as “syouken”in Japanese. The word “condition” included in the keyword matches theword “condition” in the conversion candidate word. The keyword“experiment condition” does not match the word in the text information.However, the two words constituting the keyword match the conversioncandidate words. Therefore, the search unit 65 extracts the term “tenrequirements” associated with the conversion candidate words as acorresponding part from the text information.

In the above-described example, the search method in Japanese has beendescribed. The above-described method can also be applied to searches inlanguages other than Japanese.

A specific example in which the display unit 90 displays informationwill be described below. FIG. 5 shows a screen 92 of the display unit90.

A keyword input field 300 is displayed on the screen 92. For example,the user can input a keyword to the keyword input field 300 by operatingthe operation unit. When the keyword is input to the keyword input field300, the keyword reception unit 55 receives the keyword input to thekeyword input field 300. In the example shown in FIG. 5, the term“experiment condition” is input as the keyword to the keyword inputfield 300.

The search unit 65 performs a search on the basis of a keyword receivedby the keyword reception unit 55. The search unit 65 extracts a wordmatching the keyword from words (determined words) within the textinformation and the conversion candidate word. The search unit 65identifies an utterance time point by identifying time point informationcorresponding to the word matching the keyword. The reading unit 80reads object information, image information, and text informationassociated with the same time point information corresponding to theutterance time point. The display unit 90 displays the objectinformation, the image information, and the text information read by thereading unit 80 such that the object information, the image information,and the text information are associated with each other. The objectinformation, the image information, and the text information associatedwith the same object are displayed on the screen 92. In this example,information in the observation based on the microscope system 11 shownin FIG. 3 is displayed.

In FIG. 5, two sets of the object information, the image information,and the text information are displayed. The text information isdisplayed for each appropriate block such as a sentence or a clause.Here, the corresponding object information, image information, and textinformation are displayed as the same set on the basis of the time pointinformation associated with the block. The object information, the imageinformation, and the text information corresponding to the same set arearranged in the horizontal direction. The object information, the imageinformation, and the text information of each set are arranged at thesame position in the vertical direction. The object information, theimage information, and the text information corresponding to the firstset are displayed in a region 301 of the screen 92. The objectinformation, the image information, and the text informationcorresponding to the second set are displayed in a region 302 of thescreen 92.

The object information is an image generated by a camera connected tothe microscope 200. The object information is displayed in a region 303of the screen 92. The image information is displayed in a region 304, aregion 305, and a region 306 of the screen 92. Image informationgenerated by the camera 31 a for photographing the vicinity of the tipof the objective lens of the microscope is displayed in the region 304.Image information generated by the camera 31 b for photographing all ofthe object OB1 and the microscope 200 is displayed in the region 305.Image information generated by the camera 31 c attached to the user isdisplayed in the region 306. The text information is displayed in aregion 307 of the screen 92. The text information corresponding to thefirst set includes the term “ten requirements” misrecognized withrespect to the keyword “experiment condition”. The text informationcorresponding to the second set includes text correctly recognized withrespect to the keyword “experiment condition”.

When the object information recorded on the recording medium 70 isdivided into a plurality of pieces in a time series, the reading unit 80reads the object information associated with the time point informationwithin an event period corresponding to the utterance time point fromthe recording medium 70. The display unit 90 displays the objectinformation read by the reading unit 80. For example, the objectinformation is image information of the object and the image informationof the object is moving-image information. The moving-image informationincludes image information of a plurality of frames generated atdifferent time points. In this case, the reading unit 80 reads imageinformation of the object of the plurality of frames generated duringthe event period from the recording medium 70. The display unit 90sequentially displays the image information of the object of theplurality of frames read by the reading unit 80. For example, when theuser has operated an icon 401, the display unit 90 displays a movingimage of the object during the event period. The event period will bedescribed below.

When the object information recorded on the recording medium 70 isdivided into a plurality of pieces in a time series, the reading unit 80may read representative object information associated with the timepoint information corresponding to the utterance time point from therecording medium 70. The display unit 90 may display the representativeobject information read by the reading unit 80. For example, the objectinformation is image information of the object and the image informationof the object is moving-image information. In this case, the readingunit 80 reads the image information of the object of one frame generatedat a time point closest to the utterance time point from the recordingmedium 70. The display unit 90 displays the image information of theobject of one frame read by the reading unit 80. A thumbnail of oneframe generated at a time point closest to the utterance time point maybe displayed.

When the image information recorded on the recording medium 70 isdivided into a plurality of pieces in a time series, the reading unit 80reads image information associated with the time point informationwithin the event period corresponding to the utterance time point fromthe recording medium 70. The display unit 90 displays the imageinformation read by the reading unit 80. For example, the imageinformation acquired by the image acquisition unit 30 is moving-imageinformation. In this case, the reading unit 80 reads the imageinformation of a plurality of frames generated during the event periodfrom the recording medium 70. The display unit 90 sequentially displaysthe image information of the plurality of frames read by the readingunit 80. For example, when the user has operated any one of an icon 402,an icon 403, and an icon 404, the display unit 90 displays a movingimage showing an observation situation during the event period.

When the image information recorded on the recording medium 70 isdivided into a plurality of pieces in a time series, the reading unit 80may read representative image information associated with the time pointinformation corresponding to the utterance time point from the recordingmedium 70. The display unit 90 may display the representative imageinformation read by the reading unit 80. For example, the imageinformation acquired by the image acquisition unit 30 is moving-imageinformation. The reading unit 80 reads the image information of oneframe generated at a time point closest to the utterance time point fromthe recording medium 70. The display unit 90 displays the imageinformation of one frame read by the reading unit 80. A thumbnail of oneframe generated at the time point closest to the utterance time pointmay be displayed.

The reading unit 80 reads the sound information associated with the timepoint information corresponding to the utterance time point from therecording medium 70. The sound output unit 100 outputs a sound based onthe sound information read by the reading unit 80. For example, thereading unit 80 reads the sound information associated with the timepoint information within the event period corresponding to the utterancetime point from the recording medium 70. For example, when the user hasoperated an icon 405, the sound output unit 100 outputs the sound duringthe event period.

The display unit 90 may display a word that is within the textinformation and matches the keyword such that the word is distinguishedfrom other words within the text information and display the word thatis within the text information and is associated with the conversioncandidate word matching the keyword such that the word is distinguishedfrom other words within the text information. That is, the display unit90 may display the word that is within the text information and matchesthe keyword in a first display form, display the word that is within thetext information and is associated with the conversion candidate wordmatching the keyword in a second display form, and display the otherwords within the text information in a third display form different fromthe first and second display forms. The first display form and thesecond display form may be the same. For example, the display unit 90displays the word that is within the text information and matches thekeyword and the word that is within the text information and isassociated with the conversion candidate word matching the keyword suchthat each of the displayed words is emphasized more than the other wordswithin the text information. For example, the display unit 90 displaysthe above-described words in a color different from those of the otherwords. The display unit 90 may display the above-described words withcharacters that are thicker than those of the other words. The displayunit 90 may display the above-described words with characters that arelarger than those of the other words. A method of distinguishing aspecific word from the other words is not limited to the above-describedmethod. By displaying the word corresponding to the keyword in the textinformation such that the displayed word is distinguished from otherwords, the user can easily check a search result.

Object information, image information, and text informationcorresponding to the same set may be arranged in the vertical direction.In this case, the object information, the image information, and thetext information of each set are arranged at the same position in thehorizontal direction.

The event period will be described below. The reading unit 80 reads theobject information and the image information associated with the timepoint information corresponding to the time point included in the eventperiod corresponding to the utterance time point from the recordingmedium 70. Also, the reading unit 80 reads the sound information and thetext information associated with the time point informationcorresponding to the time point included in the event periodcorresponding to the utterance time point from the recording medium 70.

FIG. 6 shows a relationship between the event occurrence time point andthe event period. An event occurrence time point T20 is a time pointfrom an event start time ta to an event end time tb. This eventcontinues to occur from the event start time ta to the event end timetb.

An event period T21 is the same as the event occurrence time point T20.An event period T22, an event period T23, and an event period T24include a time point before the event occurrence time point T20. An endpoint of the event period T22 is the event end time point tb. An endpoint of the event period T24 is the event start time point ta. An eventperiod T25, an event period T26, and an event period T27 include a timepoint later than the event occurrence time point T20. A start point ofthe event period T25 is the event start time ta. A start point of theevent period T27 is the event end time tb. An event period T28 includesonly a part of the event occurrence time point. The event period T28includes only a time point after the event start time ta and a timepoint before the event end time tb.

In the above description, at least one of a time point before the eventoccurrence time point and a time point after the event occurrence timepoint may be a preset predetermined time point. Alternatively, at leastone of the time points may be a time point relatively set on the basisof the event occurrence time point corresponding to the event period.Also, at least one of the time points may be set on the basis of theevent occurrence time point before or after the event corresponding tothe event period. The event may continue to be detected consecutivelyduring a certain period of time. Alternatively, the event may bedetected in a trigger manner for a short time. In this case, the eventoccurrence time point is approximately equal to the event end timepoint. For example, the event period may be a period from a timing thatis 5 seconds before an event in which the amplitude of the sound signalexceeds the threshold value is detected to a timing when an event inwhich an increase of the object in the image of the object is stopped isdetected.

The event period is shorter than a period from a first time point to asecond time point. The first time point is the earliest time pointindicated by the time point information associated with the objectinformation. The second time point is the latest time point indicated bythe time point information associated with the object information. Whenonly one event is detected by the event detection unit 75, the eventperiod may be the same as the period from the first time point to thesecond time point. The event period related to each piece of the imageinformation, the sound information, and the text information is similarto the event period related to the object information.

The utterance time point is a timing at which the user pays attention.As described above, information corresponding to a predetermined periodbefore or after the utterance time point is read from the recordingmedium 70. Thus, the user can efficiently view information about anevent occurring at a timing when the user pays attention.

First Modified Example of First Embodiment

FIG. 7 shows a configuration of an information recording system 10 aaccording to a first modified example of the first embodiment of thepresent invention. In terms of the configuration shown in FIG. 7,differences from the configuration shown in FIG. 1 will be described.

The information recording system 10 a includes a situation informationacquisition unit 110 in addition to the configuration of the informationrecording system 10 shown in FIG. 1. The situation informationacquisition unit 110 acquires situation information that indicates atype of situation in which the object information was acquired and isinformation other than the image information of the object. For example,the situation information is information about at least one of a timepoint, a place, and a surrounding environment of the object. Forexample, the surrounding environment of the object indicates conditionssuch as temperature, humidity, atmospheric pressure, and illuminance.When the situation information is time point information, the situationinformation acquisition unit 110 acquires the time point informationfrom a device that generates the time point information. For example,the situation information acquisition unit 110 acquires the time pointinformation from terminals such as a smartphone and a PC. When thesituation information is place information, the situation informationacquisition unit 110 acquires the place information from a device thatgenerates the place information. For example, the situation informationacquisition unit 110 acquires the place information from a terminal suchas a smartphone equipped with a Global Positioning System (GPS)function. When the situation information is surrounding environmentinformation, the situation information acquisition unit 110 acquires thesurrounding environment information from a device that measures asurrounding environment value. For example, the situation informationacquisition unit 110 acquires the surrounding environment informationfrom sensors such as a thermometer, a hygrometer, a barometer, and aluminometer.

The situation information may be device information about a deviceincluding an object information acquisition unit 20. The deviceinformation may be setting values of the device. For example, in amultiphoton excitation fluorescence microscope, the set values of thedevice are values such as lens magnification, an amount of observationlight, laser power, and a stage position. Additional information such astime point information may be added to the situation information otherthan the time point information acquired by the situation informationacquisition unit 110. For example, the situation information acquisitionunit 110 adds the time point information indicating a time point atwhich the situation information was acquired to the situationinformation and outputs the situation information to which the timepoint information is added. When the situation information istime-series information, time point information for identifying aplurality of different time points is added to the situationinformation. For example, the time point information associated with thesituation information includes a time point at which the acquisition ofthe situation information was started and a sampling rate.

A recording unit 60 records the object information, the imageinformation, the sound information, the text information, the conversioncandidate word, the situation information, and the time pointinformation on a recording medium 70 such that the object information,the image information, the sound information, the text information, theconversion candidate word, the situation information, and the time pointinformation are associated with each other. The time point informationindicates time points at which the object information, the imageinformation, the sound information, the text information, and thesituation information were acquired. The object information, the imageinformation, the sound information, the text information, the conversioncandidate word, and the situation information are associated with eachother via the time point information. The situation information may becompressed.

A reading unit 80 reads the object information, the image information,the text information, and the situation information associated with thetime point information corresponding to the word matching the keywordfrom the recording medium 70. A display unit 90 displays the objectinformation, the image information, the text information, and thesituation information read by the reading unit 80 such that the objectinformation, the image information, the text information, and thesituation information are associated with each other. For example, thedisplay unit 90 simultaneously displays the object information, theimage information, the text information, and the situation information.At this time, the display unit 90 displays the object information, theimage information, the text information, and the situation informationin a state in which these pieces of information are arranged.Information selected from the object information, the image information,the text information, and the situation information may be displayed onthe display unit 90 and the user may be able to switch information to bedisplayed on the display unit 90.

In terms of points other than the above, the configuration shown in FIG.7 is similar to the configuration shown in FIG. 1.

When the situation information is recorded, the information recordingsystem 10 a can also record other information as information indicatinga type of situation in which the object information was acquired inaddition to visual information. Thereby, the information recordingsystem 10 a can more accurately record an observation situation.Therefore, the user can more accurately reproduce and verify an accurateprocedure.

Second Modified Example of First Embodiment

FIG. 8 shows a configuration of an information recording system 10 baccording to a second modified example of the first embodiment of thepresent invention. In terms of the configuration shown in FIG. 8,differences from the configuration shown in FIG. 1 will be described.

A recording unit 60 records object information, image information, soundinformation, and time point information on a recording medium 70 suchthat the object information, the image information, the soundinformation, and the time point information are associated with eachother. A reading unit 80 reads the sound information from the recordingmedium 70. A sound processing unit 50 converts the sound informationread by a reading unit 80 into text information and generates aconversion candidate word. The recording unit 60 associates the textinformation and the conversion candidate word with the objectinformation, the image information, the sound information, and the timepoint information recorded on the recording medium 70 and records thetext information and the conversion candidate word on the recordingmedium 70. The time point information indicates a time point at whichthe sound information that is a source of the text information wasacquired.

In terms of points other than the above, the configuration shown in FIG.8 is similar to the configuration shown in FIG. 1.

In the information recording system 10 b, after the entire soundinformation is recorded on the recording medium 70, the sound processingis performed by the sound processing unit 50. Generally, the load ofsound processing is high. Even when the sound processing rate is lowerthan the acquisition rate of the sound information, the informationrecording system 10 b can record text information.

Third Modified Example of First Embodiment

FIG. 9 shows a configuration of an information recording system 10 caccording to a third modified example of the first embodiment of thepresent invention. In terms of the configuration shown in FIG. 9,differences from the configuration shown in FIG. 1 will be described.

The information recording system 10 c includes an event detection unit75 in addition to the configuration of the information recording system10 shown in FIG. 1. The event detection unit 75 detects an event on thebasis of at least one piece of the object information, the imageinformation, and the sound information recorded on the recording medium70. The event is a state in which the at least one piece of the objectinformation, the image information, and the sound information recordedon the recording medium 70 satisfies a predetermined condition. Forexample, the event detection unit 75 includes an information processingcircuit that performs information processing. When the event detectionunit 75 processes the image information, the event detection unit 75includes an image processing circuit. When the event detection unit 75processes the sound information, the event detection unit 75 includes asound processing circuit. For example, the at least one piece of theobject information, the image information, and the sound informationrecorded on the recording medium 70 is read by the reading unit 80. Theevent detection unit 75 detects an event on the basis of the informationread by the reading unit 80. Also, the time point information recordedon the recording medium 70 is read by the reading unit 80. The eventdetection unit 75 identifies the event occurrence time point which is atime point at which the event occurred on the basis of a relationshipbetween the time point information read by the reading unit 80 andinformation in which the event was detected. The recording unit 60 mayrecord the event occurrence time point identified by the event detectionunit 75 on the recording medium 70.

The reading unit 80 reads the object information, the image information,and the text information associated with the time point informationcorresponding to the event occurrence time point from the recordingmedium 70. The display unit 90 displays the object information, theimage information, and the text information read by the reading unit 80.The search unit 65 searches for a keyword received by a keywordreception unit 55 in the text information and a conversion candidateword associated with the time point information corresponding to theevent occurrence time point. The display unit 90 displays the objectinformation, the image information, and the text information associatedwith the time point information corresponding to the event occurrencetime point at which the word matching the keyword was detected such thatthe object information, the image information, and the text informationare distinguished from other object information, other imageinformation, and other text information. That is, the display unit 90displays the object information, the image information, and the textinformation associated with the time point information corresponding tothe event occurrence time point at which the word matching the keywordwas detected in a first display form and displays other objectinformation, other image information, and other text information in asecond display form different from the first display form. The displayforms are distinguished by changing a color and a thickness of a framefor displaying information, for example, according to a degree ofemphasis for display.

The time point information is added to a determined word constitutingthe text information. Alternatively, the time point information is addedto the determined word and the conversion candidate word. The eventoccurrence time point at which the word matching the keyword wasdetected is a time point indicated by the time point information addedto the word. The conversion candidate word is associated with thedetermined word constituting the text information. When the conversioncandidate word to which the time point information is not added matchesthe keyword, the time point information added to the determined wordassociated with the conversion candidate word indicates the eventoccurrence time point at which the conversion candidate word wasdetected.

The time point information may be added to the text information for eachevent detected by the event detection unit 75. That is, the eventoccurrence time point may be added to the text information.

When the information recording system 10 c does not include the imageacquisition unit 30, the recording unit 60 records the objectinformation, the sound information, the text information, the conversioncandidate word, and the time point information on the recording medium70 such that the object information, the sound information, the textinformation, the conversion candidate word, and the time pointinformation are associated with each other. The event detection unit 75detects an event on the basis of at least one piece of the objectinformation and the sound information recorded on the recording medium70. The reading unit 80 reads the object information, the soundinformation, and the text information associated with the time pointinformation corresponding to the event occurrence time point from therecording medium 70. The display unit 90 displays the object informationand the text information read by the reading unit 80 such that theobject information and the text information are associated with eachother. The sound output unit 100 outputs a sound on the basis of thesound information read by the reading unit 80. The display unit 90displays the object information and the text information associated withthe time point information corresponding to the event occurrence timepoint at which the word matching the keyword was detected such that theobject information and the text information are distinguished from otherobject information and other text information.

The information recording system 10 c may not include the sound outputunit 100 and the recording unit 60 may not record sound information. Inthis case, the recording unit 60 records the object information, theimage information, the text information, the conversion candidate word,and the time point information on the recording medium 70 such that theobject information, the image information, the text information, theconversion candidate word, and the time point information are associatedwith each other. The event detection unit 75 detects an event on thebasis of at least one piece of the object information and the imageinformation recorded on the recording medium 70. The reading unit 80reads the object information, the image information, and the textinformation associated with the time point information corresponding tothe event occurrence time point from the recording medium 70. Thedisplay unit 90 displays the object information, the image information,and the text information read by the reading unit 80 such that theobject information, the image information, and the text information areassociated with each other. The display unit 90 displays the objectinformation, the image information, and the text information associatedwith the time point information corresponding to the event occurrencetime point at which the word matching the keyword was detected such thatthe object information, the image information, and the text informationare distinguished from other object information, other imageinformation, and other text information.

In terms of points other than the above, the configuration shown in FIG.9 is similar to the configuration shown in FIG. 1.

FIG. 10 shows a procedure of processing of the information recordingsystem 10 c. In terms of the processing shown in FIG. 10, differencesfrom the processing shown in FIG. 2 will be described.

After step S115, the event detection unit 75 detects an event on thebasis of at least one piece of object information, image information,and sound information recorded on the recording medium 70 (step S140 (anevent detection step)).

After step S140, the reading unit 80 reads the object information, theimage information, the sound information, and text informationassociated with time point information corresponding to an eventoccurrence time point that is a time point at which an event occurredfrom the recording medium 70 (step S145 (a reading step)). The user maybe able to specify a timing at which the information is read.

After step S145, the display unit 90 displays the object information,the image information, and the text information read by the reading unit80 such that the object information, the image information, and the textinformation are associated with each other. Also, the sound output unit100 outputs a sound based on the sound information read by the readingunit 80 (step S150 (a display step and a sound output step)). After stepS150, the processing in step S120 is performed.

After step S125, the display unit 90 displays the object information,the image information, and the text information associated with the timepoint information corresponding to the event occurrence time point atwhich a word matching a keyword was detected such that the objectinformation, the image information, and the text information aredistinguished from other object information, other image information,and other text information (step S155 (a display step)).

In terms of points other than the above, the processing shown in FIG. 10is similar to the processing shown in FIG. 2.

When the information recording system 10 c does not include the imageacquisition unit 30, the processing in step S105 is not performed. Also,in step S115, the recording unit 60 records the object information, thesound information, the text information, the conversion candidate word,and the time point information on the recording medium 70 such that theobject information, the sound information, the text information, theconversion candidate word, and the time point information are associatedwith each other. In step S140, the event detection unit 75 detects anevent on the basis of at least one piece of the object information andthe sound information recorded on the recording medium 70. In step S145,the reading unit 80 reads the object information, the sound information,and the text information associated with the time point informationcorresponding to the event occurrence time point from the recordingmedium 70. In step S150, the display unit 90 displays the objectinformation and the text information read by the reading unit 80 suchthat the object information and the text information are associated witheach other. Also, in step S150, the sound output unit 100 outputs asound based on the sound information read by the reading unit 80. Instep S155, the display unit 90 displays the object information and thetext information associated with the time point informationcorresponding to the event occurrence time point at which the wordmatching the keyword was detected such that the object information andthe text information are distinguished from other object information andother text information.

When the information recording system 10 c does not include the soundoutput unit 100 and the recording unit 60 does not record soundinformation, the recording unit 60 records object information, imageinformation, text information, a conversion candidate word, and timepoint information on the recording medium 70 such that the objectinformation, the image information, the text information, the conversioncandidate word, and the time point information are associated with eachother in step S115. In step S140, the event detection unit 75 detects anevent on the basis of at least one piece of the object information andthe image information recorded on the recording medium 70. In step S145,the reading unit 80 reads the object information, the image information,and the text information associated with the time point informationcorresponding to the event occurrence time point from the recordingmedium 70. In step S150, the display unit 90 displays the objectinformation, the image information, and the text information read by thereading unit 80 such that the object information, the image information,and the text information are associated with each other. In step S155,the display unit 90 displays the object information, the imageinformation, and the text information associated with the time pointinformation corresponding to the event occurrence time point at whichthe word matching the keyword was detected such that the objectinformation, the image information, and the text information aredistinguished from other object information, other image information,and other text information.

As described above, the event is detected on the basis of at least onepiece of the object information, the image information, and the soundinformation recorded on the recording medium 70. Information associatedwith the time point information corresponding to the event occurrencetime point at which the word matching the keyword within the informationcorresponding to the event was detected is displayed such that theinformation is distinguished from other information. Thereby, theinformation recording system 10 c can support efficient informationviewing of the user.

According to the above-described method, the information recordingsystem 10 c can extract a useful scene to which the user pays attentionand a list of information about the scene from a plurality of pieces ofand a large amount of information recorded at an observation site. Also,the information recording system 10 c can extract informationcorresponding to a keyword from the list. Therefore, the user canefficiently view information about an event occurring at a timing whenthe user pays attention.

A specific example of event detection of the event detection unit 75will be described below.

FIGS. 11 and 12 show examples of event detection based on the objectinformation. In FIGS. 11 and 12, the object information is imageinformation (a microscope image) of the object acquired by the cameraconnected to the microscope.

As shown in FIG. 11, an object OB10 is included in an image G10. Animage G11 is captured at a time point later than that when the image G10was captured. The object OB10 is included in the image G11. The shape ofthe object OB10 is different between the image G10 and the image G11.That is, the shape of the object OB10 varies with time. When the shapeof the object OB10 has changed, the event detection unit 75 detects anevent. For example, the event detection unit 75 determines whether ornot an event in which the shape of the object OB10 changes has occurredby comparing image information of a plurality of frames acquired atdifferent time points.

As shown in FIG. 12, an object OB11, an object OB12, an object OB13, andan object OB14 are included in an image G12. An image G13 is captured ata time point later than a time point when the image G12 was captured. Inaddition to the objects OB11 to OB14, an object OB15, an object OB16,and an object OB17 are included in the image G13. The objects OB15 toOB17 are added between the image G12 and the image G13. That is, thenumber of objects varies with time. When the number of objects haschanged, the event detection unit 75 detects an event. For example, theevent detection unit 75 determines whether or not an event in which thenumber of objects changes has occurred by comparing image information ofa plurality of frames acquired at different time points.

When a state of the object indicated by the object information is astate predefined as an event detection condition, the event detectionunit 75 detects the event. For example, the event detection condition isrecorded on the recording medium 70 in advance. The reading unit 80reads the event detection condition from the recording medium 70. Theevent detection unit 75 detects an event on the basis of the eventdetection condition read by the reading unit 80. Thereby, the eventdetection unit 75 can detect a phenomenon that the object is in apredetermined state as the event.

The image acquisition unit 30 acquires image information including animage of at least one of the object and surroundings of the object. Whenat least one state of the object and the surroundings of the objectindicated by the image information is a state predefined as the eventdetection condition, the event detection unit 75 detects the event. Forexample, when a feature of the image information is the same as afeature predefined as the event detection condition, the event detectionunit 75 detects the event. For example, in a microscope system using amultiphoton excitation fluorescence microscope, the event detection unit75 detects an event when it is detected that light has entered a darkroom from the image information. For example, in the examination system13, the event detection unit 75 detects an event when a state such asbleeding or seizure of a patient is detected from image information. Forexample, feature information indicating the above-described feature isrecorded on the recording medium 70 in advance as an event detectioncondition. The event detection unit 75 extracts the feature informationfrom the image information. The reading unit 80 reads the eventdetection condition from the recording medium 70. The event detectionunit 75 compares the feature information extracted from the imageinformation with the feature information that is the event detectioncondition read by the reading unit 80. When the feature informationextracted from the image information is the same as or similar to thefeature information as the event detection condition, the eventdetection unit 75 detects the event. Thereby, the event detection unit75 can detect a phenomenon that the observation state indicated by theimage information becomes a predetermined state as an event.

FIGS. 13 and 14 show examples of event detection based on soundinformation. In FIGS. 13 and 14, the sound information is a time-seriessound signal (sound data). The sound signal includes amplitudeinformation of the sound at each of the plurality of time points. FIG.13 shows a graph of a sound signal A10 and FIG. 14 shows a graph of asound signal A11. In the graphs of FIGS. 13 and 14, the horizontaldirection represents time and the vertical direction representsamplitude.

The sound signal A10 shown in FIG. 13 is a sound during an inspectionwith an industrial endoscope. For example, the amplitude of the soundsignal exceeds a threshold value during a period T10, a period T11, anda period T12 shown in FIG. 13. The threshold value is greater than zero.The user is an inspector. For example, during the period T10, the userutters a sound indicating that there is a scratch at a position of 250mm. For example, during the period T11, the user utters a soundindicating that there is a hole with a diameter of 5 mm at a position of320 mm. For example, during the period T12, the user utters a soundindicating that there is rust at a position of 470 mm. When theamplitude of the sound signal exceeds the predetermined threshold value,the event detection unit 75 detects the event. Even when the user hasuttered a series of sounds, a sound signal at that time includes aperiod with small amplitude. When a plurality of events are continuouslydetected within a predetermined time, the event detection unit 75 mayaggregate the plurality of events as one event. Alternatively, the eventdetection unit 75 may use an average value of amplitudes within apredetermined time or the like as a representative value and detect thepresence or absence of an event at predetermined time intervals. In thismanner, the event detection unit 75 detects the event during the periodT10, the period T11, and the period T12 corresponding to a period duringwhich the user has uttered the sound.

The threshold value may be smaller than zero. If the amplitude of thesound signal is smaller than the threshold value smaller than 0, theamplitude of the sound signal exceeds the threshold value. When thepower of the sound signal exceeds a predetermined threshold value, theevent detection unit 75 may detect the event. For example, the power ofthe sound signal is a square mean value of the amplitude.

As described above, the sound acquisition unit 40 acquires soundinformation based on the sound uttered by the observer who observes theobject. The sound information is a time-series sound signal. When theamplitude or the power of the sound signal exceeds the threshold valuepredefined as the event detection condition, the event detection unit 75detects the event. For example, a threshold value determined on thebasis of amplitude or power of predetermined sound information or athreshold value specified by a user who is the observer is recorded onthe recording medium 70 in advance as the event detection condition. Thereading unit 80 reads the event detection condition from the recordingmedium 70. The event detection unit 75 compares the amplitude or powerof the sound signal acquired by the sound acquisition unit 40 with thethreshold value which is the event detection condition read by thereading unit 80. When the amplitude or the power of the sound signalexceeds the threshold value, the event detection unit 75 detects theevent. Thereby, the event detection unit 75 can detect a phenomenon whenthe user has uttered comments as the event.

The sound signal A11 shown in FIG. 14 is a sound during the inspectionwith a medical endoscope. The user is a doctor. For example, during theperiod T13 shown in FIG. 14, the user utters a word “polyp”. When theword “polyp” is registered as a keyword for event detection in advance,the event detection unit 75 detects an event during the period T13.

As described above, the sound acquisition unit 40 acquires soundinformation based on the sound uttered by the observer who observes theobject. When the sound indicated by the sound information is the same asthe sound of the keyword predefined as the event detection condition,the event detection unit 75 detects the event. For example, the soundinformation generated by acquiring the sound of the keyword is recordedon the recording medium 70 in advance as the event detection condition.The reading unit 80 reads the event detection condition from therecording medium 70. The event detection unit 75 compares the soundinformation acquired by the sound acquisition unit 40 with the soundinformation that is the event detection condition read by the readingunit 80. For example, when the two pieces of the sound information arethe same, i.e., when the similarity between the two pieces of the soundinformation is greater than or equal to a predetermined value, the eventdetection unit 75 detects the event. Thereby, the event detection unit75 can detect a phenomenon when the user who is the observer utters apredetermined keyword as the event.

In the observation site, in many cases, the user recognizes the state ofthe object or the observation situation and utters comments with respectto the state of the object or the observation situation. Thus, when theevent detection unit 75 detects an event on the basis of the soundinformation, the event detection unit 75 can more easily detect theevent to which the user pays attention.

A specific example of display of information by the display unit 90 willbe described below. FIG. 15 shows a window W10 displayed on a screen 93of the display unit 90.

The event detection unit 75 detects an event on the basis of at leastone piece of the object information, the image information, and thesound information recorded on the recording medium 70. The reading unit80 reads the object information, the image information, and the textinformation associated with the time point information corresponding tothe event occurrence time point from the recording medium 70. Thedisplay unit 90 displays the object information, the image information,and the text information read by the reading unit 80 such that theobject information, the image information, and the text information areassociated with each other.

The event detection unit 75 detects a plurality of event occurrence timepoints. The reading unit 80 reads the object information, the imageinformation, and the text information associated with time pointinformation corresponding to each of the plurality of event occurrencetime points from the recording medium 70. The display unit 90 displaysthe object information, the image information, and the text informationread by the reading unit 80 at each event occurrence time point suchthat the object information, the image information, and the textinformation are associated with each other. In the present example,information in the observation with the microscope system 11 shown inFIG. 3 is shown.

In FIG. 15, object information, image information, and text informationcorresponding to five events among a plurality of events are shown.Object information, image information, and text informationcorresponding to the same event are displayed such that the objectinformation, the image information, and the text information areassociated with each other. The object information, the imageinformation, and the text information corresponding to the same eventare arranged in the horizontal direction. The object information, theimage information, and the text information in each event are associatedwith the same position of the vertical direction. The objectinformation, the image information, and the text informationcorresponding to an event 1 are displayed in a region 308 of the screen93. The object information, the image information, and the textinformation corresponding to an event 2 are displayed in a region 309 ofthe screen 93. The object information, the image information, and thetext information corresponding to an event 3 are displayed in a region310 of the screen 93. The object information, the image information, andthe text information corresponding to an event 4 are displayed in aregion 311 of the screen 93. The object information, the imageinformation, and the text information corresponding to an event 5 aredisplayed in a region 312 of the screen 93.

The object information is an image generated by a camera connected tothe microscope 200. The object information is displayed in a region 313of the screen 93. The image information is displayed in a region 314, aregion 315, and a region 316 of the screen 93. Image informationgenerated by the camera 31 a for photographing the vicinity of the tipof the objective lens of the microscope is displayed in the region 314.Image information generated by the camera 31 b for photographing all ofthe object OB1 and the microscope 200 is displayed in the region 315.Image information generated by the camera 31 c attached to the user isdisplayed in the region 316. Text information is displayed in a region317 of the screen 93.

When the object information recorded on the recording medium 70 isdivided into a plurality of pieces in a time series, the reading unit 80reads object information associated with the time point informationwithin the event period corresponding to the event occurrence time pointfrom the recording medium 70. The display unit 90 displays the objectinformation read by the reading unit 80. For example, when the user hasoperated an icon 401, the display unit 90 displays a moving image of theobject during the event period. The reading unit 80 may readrepresentative object information associated with the time pointinformation corresponding to the event occurrence time point from therecording medium 70. The display unit 90 may display the representativeobject information read by the reading unit 80. For example, therepresentative object information is image information of an object ofone frame generated at a time point closest to the event occurrence timepoint. The representative object information may be a thumbnail of oneframe generated at the time point closest to the event occurrence timepoint.

When the image information recorded on the recording medium 70 isdivided into a plurality of pieces in a time-series, the reading unit 80reads the image information associated with the time point informationwithin the event period corresponding to the event occurrence time pointfrom the recording medium 70. The display unit 90 displays the imageinformation read by the reading unit 80. For example, when the user hasoperated any one of an icon 402, an icon 403, and an icon 404, thedisplay unit 90 displays a moving image showing an observation situationduring the event period. The reading unit 80 may read representativeimage information associated with the time point informationcorresponding to the event occurrence time point from the recordingmedium 70. The display unit 90 may display representative imageinformation read by the reading unit 80. For example, the representativeimage information is image information of one frame generated at thetime point closest to the event occurrence time point. Therepresentative image information may be a thumbnail of one framegenerated at the time point closest to the event occurrence time point.

The reading unit 80 reads the sound information associated with the timepoint information corresponding to the event occurrence time point fromthe recording medium 70. The sound output unit 100 outputs a sound basedon the sound information read by the reading unit 80. For example, thereading unit 80 reads the sound information associated with the timepoint information within the event period corresponding to the eventoccurrence time point from the recording medium 70. For example, whenthe user has operated an icon 405, the sound output unit 100 outputs asound during the event period.

The description of the event period in the first embodiment can beapplied to the description of the event period in the third modifiedexample of the first embodiment. An utterance time point in the abovedescription is replaced with the event occurrence time point.

After the information of each event shown in FIG. 15 is displayed, theuser can input a keyword to a keyword input field 300 by operating anoperation unit. When the keyword is input to the keyword input field300, the keyword reception unit 55 receives the keyword input to thekeyword input field 300. In the example shown in FIG. 15, the term“experiment condition” is input as the keyword to the keyword inputfield 300.

The search unit 65 searches for the keyword received by the keywordreception unit 55 in the text information and the conversion candidateword associated with the time point information within the event period.That is, the search unit 65 searches for the keyword in the textinformation displayed by the display unit 90 and the conversioncandidate word associated with the word within the text information. Thesearch unit 65 extracts the word matching the keyword from the word (thedetermined word) within the text information and the conversioncandidate word. The display unit 90 displays the object information, theimage information, and the text information within an event periodcorresponding to the time point information corresponding to an eventoccurrence time point at which the word matching the keyword wasdetected such that the object information, the image information, andthe text information are distinguished from other object information,other image information, and other text information.

In the example shown in FIG. 15, the text information corresponding tothe event 1 does not include the keyword “experiment condition”, butincludes the term “ten requirements” which is an erroneous recognitionresult. Also, conversion candidate words associated with the textinformation corresponding to the event 1 include the keyword “experimentcondition”. The text information corresponding to the event 3 includesthe keyword “experiment condition”. Thus, the display unit 90 displaysthe object information, the image information, and the text informationcorresponding to each of the event 1 and the event 3 such that they aredistinguished from the object information, the image information, andthe text information corresponding to each of the event 2, the event 4,and the event 5. Specifically, the display unit 90 displays the objectinformation, the image information, and the text informationcorresponding to each of the event 1 and the event 3 such that they areemphasized more than the object information, the image information, andthe text information corresponding to each of the event 2, the event 4,and the event 5. For example, the object information, the imageinformation, and the text information corresponding to each of the event1 and the event 3 are surrounded by thick lines.

The display unit 90 may brightly display the object information, theimage information, and the text information corresponding to each of theevent 1 and the event 3. The display unit 90 may enlarge and display theobject information, the image information, and the text informationcorresponding to each of the event 1 and the event 3. A method ofdistinguishing information of an event corresponding to an eventoccurrence time point at which a word matching a keyword was detectedfrom information of another event is not limited to the above-describedmethod.

The display unit 90 may display the word that is within the textinformation and matches the keyword such that the word is distinguishedfrom other words within the text information and display the word thatis within the text information and is associated with the conversioncandidate word matching the keyword such that the word is distinguishedfrom other words within the text information.

As described above, information associated with the time pointinformation corresponding to the event occurrence time point at whichthe word matching the keyword within the information corresponding tothe event was detected is displayed such that the information isdistinguished from other information. Thus, the user can easilyrecognize information corresponding to a desired keyword. Also, the wordcorresponding to the keyword in the text information is displayed suchthat the word is distinguished from other words and therefore the usercan easily check a search result.

The information recording system 10 c may include a situationinformation acquisition unit 110 in the information recording system 10a shown in FIG. 7. When the situation indicated by situation informationis a state predefined as an event detection condition, the eventdetection unit 75 may detect an event. For example, the situationinformation is surrounding environment information acquired from athermometer, i.e., a temperature. When the temperature indicated by thesituation information exceeds a threshold value predefined as the eventdetection condition, the event detection unit 75 detects an event. Forexample, a threshold value specified by the user is recorded on therecording medium 70 in advance as the event detection condition. Thereading unit 80 reads the event detection condition from the recordingmedium 70. The event detection unit 75 compares the temperatureindicated by the situation information acquired by the situationinformation acquisition unit 110 with the threshold value that is theevent detection condition read by the reading unit 80. When thetemperature exceeds the threshold value, the event detection unit 75detects an event.

Second Embodiment

FIG. 16 shows a configuration of an information recording system 10 daccording to a second embodiment of the present invention. In terms ofthe configuration shown in FIG. 16, differences from the configurationshown in FIG. 1 will be described.

As shown in FIG. 16, the information recording system 10 d includes anobject information acquisition unit 20, an image acquisition unit 30, asound acquisition unit 40, an information recording device 120, adisplay unit 90, and a sound output unit 100. Configurations of theobject information acquisition unit 20, the image acquisition unit 30,the sound acquisition unit 40, the display unit 90, and the sound outputunit 100 are similar to those corresponding to the components shown inFIG. 1. In the information recording system 10 d shown in FIG. 16, thesound processing unit 50, the keyword reception unit 55, the recordingunit 60, the search unit 65, the recording medium 70, and the readingunit 80 in the information recording system 10 shown in FIG. 1 arechanged to the information recording device 120.

In terms of points other than the above, the configuration shown in FIG.16 is similar to the configuration shown in FIG. 1.

FIG. 17 shows a configuration of the information recording device 120.As shown in FIG. 17, the information recording device 120 includes asound processing unit 50, a keyword reception unit 55, a recording unit60, a search unit 65, a recording medium 70, a reading unit 80, an inputunit 130, and an output unit 140.

The configurations of the sound processing unit 50, the keywordreception unit 55, the recording unit 60, the search unit 65, therecording medium 70, and the reading unit 80 are similar to thosecorresponding to the components shown in FIG. 1. Object information fromthe object information acquisition unit 20, image information from theimage acquisition unit 30, and sound information from the soundacquisition unit 40 are input to the input unit 130. For example, atleast one of the object information acquisition unit 20, the imageacquisition unit 30, and the sound acquisition unit 40 is connected tothe information recording device 120 through a cable. In this case, theinput unit 130 is an input terminal to which the cable is connected. Atleast one of the object information acquisition unit 20, the imageacquisition unit 30, and the sound acquisition unit 40 may be wirelesslyconnected to the information recording device 120. In this case, theinput unit 130 is a wireless communication circuit that wirelesslycommunicates with at least one of the object information acquisitionunit 20, the image acquisition unit 30, and the sound acquisition unit40.

The output unit 140 outputs the object information, the imageinformation, the sound information, and the text information read by thereading unit 80. That is, the output unit 140 outputs the objectinformation, the image information, and the text information to thedisplay unit 90 and outputs the sound information to the sound outputunit 100. For example, at least one of the display unit 90 and the soundoutput unit 100 is connected to the information recording device 120through a cable. In this case, the output unit 140 is an output terminalto which the cable is connected. At least one of the display unit 90 andthe sound output unit 100 may be wirelessly connected to the informationrecording device 120. In this case, the output unit 140 is a wirelesscommunication circuit that wirelessly communicates with at least one ofthe display unit 90 and the sound output unit 100.

The information recording device 120 may read a program and execute theread program. That is, the function of the information recording device120 may be implemented by software. This program includes instructionsfor defining the operations of the sound processing unit 50, therecording unit 60, the search unit 65, and the reading unit 80. Forexample, this program may be provided by a “computer-readable recordingmedium” such as a flash memory. Also, the above-described program may betransmitted from a computer having a storage device or the like in whichthe program is stored to the information recording device 120 via atransmission medium or transmission waves in the transmission medium.The “transmission medium” for transmitting the program refers to amedium having an information transmission function, for example, anetwork (a communication network) such as the Internet or acommunication circuit (a communication line) such as a telephonecircuit. Also, the above-described program may be a program forimplementing some of the above-described functions. Further, theabove-described program may be a program capable of implementing theabove-described function in combination with a program already recordedon the computer, i.e., a so-called differential file (differentialprogram).

Various modifications applied to the information recording system 10shown in FIG. 1 may be similarly applied to the information recordingsystem 10 d shown in FIG. 16. For example, the information recordingsystem 10 d may not include the image acquisition unit 30. In this case,the object information and the sound information are input to the inputunit 130. The recording unit 60 records the object information, thesound information, the text information, the conversion candidate word,and the time point information on the recording medium 70 such that theobject information, the sound information, the text information, theconversion candidate word, and the time point information are associatedwith each other. The reading unit 80 reads the object information andthe text information associated with the time point informationcorresponding to the word matching the keyword from the recording medium70. The output unit 140 outputs the object information, the soundinformation, and the text information read by the reading unit 80. Thedisplay unit 90 displays the object information and the text informationoutput by the output unit 140 such that the object information and thetext information are associated with each other. The text informationmay not be output from the information recording device 120. The soundoutput unit 100 outputs a sound based on the sound information output bythe output unit 140.

The information recording system 10 d does not include the sound outputunit 100 and the recording unit 60 may not record sound information. Inthis case, the object information, the image information, and the soundinformation are input to the input unit 130. The recording unit 60records the object information, the image information, the textinformation, the conversion candidate word, and the time pointinformation on the recording medium 70 such that the object information,the image information, the text information, the conversion candidateword, and the time point information are associated with each other. Thereading unit 80 reads the object information, the image information, andthe text information associated with the time point informationcorresponding to the word matching the keyword from the recording medium70. The output unit 140 outputs the object information, the imageinformation, and the text information read by the reading unit 80. Thedisplay unit 90 displays the object information, the image information,and the text information output by the output unit 140 such that theobject information, the image information, and the text information areassociated with each other. No text information may be output from theinformation recording device 120.

FIG. 18 shows a procedure of processing of the information recordingdevice 120. The procedure of processing of the information recordingdevice 120 will be described with reference to FIG. 18.

Object information about the object is input to the input unit 130 (stepS200 (an input step)). The object information input in step S200 isstored in a buffer within the recording unit 60. In parallel with theinput of the object information to the input unit 130, image informationindicating a type of situation in which the object information wasacquired is input to the input unit 130 (step S205 (an input step)). Theimage information input in step S205 is stored in the buffer within therecording unit 60. In parallel with the input of the object informationto the input unit 130, the processing in step S210 is performed. StepS210 includes step S211 (a sound input step) and step S212 (a soundprocessing step). In step S211, sound information based on a sounduttered by an observer who observes the object is input to the inputunit 130. In step S212, the sound processing unit 50 converts the soundinformation input to the input unit 130 into text information andgenerates at least one conversion candidate word corresponding to thesound in the process of converting the sound information into the textinformation. In step S210, the processing in steps S211 and S212 isiterated. The sound information input in step S211, the text informationgenerated in step S212, and the conversion candidate word generated instep S212 are stored in the buffer within the recording unit 60.

Processing start timings of step S200, step S205, and step S210 may notbe the same. Processing end timings of step S200, step S205, and stepS210 may not be the same. At least some of periods during which theprocessing in step S200, step S205, and step S210 is performed overlapeach other.

After the input of the object information, the image information, andthe sound information is completed, the recording unit 60 records theobject information, the image information, the sound information, thetext information, the conversion candidate word, and the time pointinformation stored in the buffer within the recording unit 60 on therecording medium 70 such that the object information, the imageinformation, the sound information, the text information, the conversioncandidate word, and the time point information are associated with eachother (step S215 (a recording step)).

After step S215, the keyword reception unit 55 receives a keyword (stepS220 (a keyword reception step)).

After step S220, the search unit 65 searches for the keyword received bythe keyword reception unit 55 in the text information and the conversioncandidate word, and extracts a word matching the keyword from wordswithin the text information and the conversion candidate word (step S225(a search step)).

After step S225, the reading unit 80 reads the object information, theimage information, the sound information, and the text informationassociated with the time point information corresponding to the wordmatching the keyword from the recording medium 70 (step S230 (a readingstep)). The user may be able to specify a timing at which theinformation is read.

After step S230, the output unit 140 outputs the object information, theimage information, the sound information, and the text information readby the reading unit 80. The display unit 90 displays the objectinformation, the image information, and the text information output bythe output unit 140 such that the object information, the imageinformation, and the text information are associated with each other.Also, the sound output unit 100 outputs a sound based on the soundinformation output by the output unit 140 (step S235 (an output step, adisplay step, and a sound output step)).

When the information recording system 10 d does not include the imageacquisition unit 30, the processing in step S205 is not performed. Also,in step S215, the recording unit 60 records the object information, thesound information, the text information, the conversion candidate word,and the time point information on the recording medium 70 such that theobject information, the sound information, the text information, theconversion candidate word, and the time point information are associatedwith each other. In step S230, the reading unit 80 reads the objectinformation, the sound information, and the text information associatedwith the time point information corresponding to the word matching thekeyword from the recording medium 70. In step S235, the output unit 140outputs the object information, the sound information, and the textinformation read by the reading unit 80 in step S230. Also, in stepS235, the display unit 90 displays the object information and the textinformation output by the output unit 140 such that the objectinformation and the text information are associated with each other. Notext information may be output from the information recording device120. Also, in step S235, the sound output unit 100 outputs a sound basedon the sound information read by the reading unit 80 in step S230.

When the information recording system 10 d does not include the soundoutput unit 100 and the recording unit 60 does not record soundinformation, the recording unit 60 records the object information, theimage information, the text information, the conversion candidate word,and the time point information on the recording medium 70 such that theobject information, the image information, the text information, theconversion candidate word, and the time point information are associatedwith each other in step S215. In step S230, the reading unit 80 readsthe object information, the image information, and the text informationassociated with the time point information corresponding to the wordmatching the keyword from the recording medium 70. In step S235, theoutput unit 140 outputs the object information, the image information,and the text information read by the reading unit 80 in step S230. Also,in step S235, the display unit 90 displays the object information, theimage information, and the text information output by the output unit140 such that the object information, the image information, and thetext information are associated with each other. No text information maybe output from the information recording device 120.

At least one of the sound processing unit 50 and the recording medium 70may be disposed outside the information recording device 120. When thesound processing unit 50 is disposed outside the information recordingdevice 120, the text information from the sound processing unit 50 isinput to the input unit 130. The recording medium 70 may be attachableto and detachable from the information recording device 120. Theinformation recording device 120 may have a network interface and theinformation recording device 120 may be connected to the recordingmedium 70 via a network. The information recording device 120 may have awireless communication interface and the information recording device120 may be connected to the recording medium 70 through wirelesscommunication.

The information recording device 120 may not include the output unit140. For example, the recording medium 70 is configured so that therecording medium 70 can be attached to and detached from the informationrecording device 120. The reading unit 80 reads the object information,the image information, the sound information, and the text informationassociated with the time point information corresponding to the wordmatching the keyword from the recording medium 70. The recording unit 60records the object information, the image information, the soundinformation, and the text information read by the reading unit 80 on therecording medium 70 such that the object information, the imageinformation, the sound information, and the text information areassociated with each other. When the recording medium 70 is detachedfrom the information recording device 120 and is attached to a deviceoutside the information recording device 120, the device can use theinformation recorded on the recording medium 70. When the informationrecording device 120 does not include the output unit 140, theinformation recording device 120 does not perform the processing in stepS235.

As described above, object information is input to the input unit 130and image information indicating a type of situation in which the objectinformation was acquired is input to the input unit 130. The inputobject information and image information are recorded on the recordingmedium 70 by the recording unit 60. Thereby, the information recordingdevice 120 can record visual information indicating a type of situationin which the object information was acquired.

As described above, the conversion candidate word generated in theprocess of converting the sound information into the text information isrecorded on the recording medium 70 by the recording unit 60. The searchunit 65 searches for a keyword received by the keyword reception unit 55in the text information and the conversion candidate word and extracts aword matching the keyword from the words within the text information andthe conversion candidate word. Even when a word erroneously recognizedin the sound recognition process cannot be extracted from the textinformation during a search, the search unit 65 can extract a desiredword from the conversion candidate word corresponding to the erroneouslyrecognized word. Therefore, the information recording device 120 canreduce omissions in a word search in the text information. As a result,the information recording device 120 can provide the user with theobject information, the image information, the sound information, andthe text information corresponding to the keyword received by thekeyword reception unit 55. The effect obtained in the informationrecording system 10 of the first embodiment can be similarly obtainedalso in the information recording device 120 of the second embodiment.

The matters disclosed in the first to third modified examples of thefirst embodiment may be similarly applied to the information recordingdevice 120 of the second embodiment. Therefore, the informationrecording system 10 d may include the situation information acquisitionunit 110 and the situation information acquired by the situationinformation acquisition unit 110 may be input to the input unit 130.Alternatively, the information recording device 120 may include an eventdetection unit 75.

Third Embodiment

FIG. 19 shows a configuration of an information recording system 10 eaccording to a third embodiment of the present invention. In terms ofthe configuration shown in FIG. 19, differences from the configurationshown in FIG. 1 will be described.

The information recording system 10 e shown in FIG. 19 does not includethe object information acquisition unit 20 and the image acquisitionunit 30 in the information recording system 10 shown in FIG. 1. Arecording unit 60 records sound information, text information,conversion candidate word, and time point information on a recordingmedium 70 such that the sound information, the text information, theconversion candidate word, and the time point information are associatedwith each other. A reading unit 80 reads text information including afirst part corresponding to a word matching the keyword from therecording medium 70. A display unit 90 displays the text informationread by the reading unit 80 such that the first part and a second partother than the first part can be distinguished.

Also, the reading unit 80 reads the sound information associated withthe time point information corresponding to the word matching thekeyword from the recording medium 70. A sound output unit 100 outputs asound based on the sound information read by the reading unit 80.

The information recording system 10 e may not include the sound outputunit 100. In this case, the recording unit 60 records the textinformation and the conversion candidate word on the recording medium 70such that the text information and the conversion candidate word areassociated with each other. The reading unit 80 reads text informationincluding the first part corresponding to the word matching the keywordfrom the recording medium 70. Because the conversion candidate word isassociated with a determined word constituting the text information, notime point information may be recorded.

In terms of points other than the above, the configuration shown in FIG.19 is similar to the configuration shown in FIG. 1.

FIG. 20 shows a procedure of processing of the information recordingsystem 10 c. In terms of the processing shown in FIG. 20, differencesfrom the processing shown in FIG. 2 will be described.

The processing in steps S100 and S105 shown in FIG. 2 is not performed.After step S110, the recording unit 60 records sound information, textinformation, conversion candidate word, and time point informationstored in the buffer in the recording unit 60 on the recording medium 70such that the sound information, the text information, the conversioncandidate word, and the time point information are associated with eachother (step S160 (a recording step)).

After step S125, the reading unit 80 reads the text informationincluding a first part corresponding to a word matching a keyword fromthe recording medium 70. Also, the reading unit 80 reads the soundinformation associated with the time point information corresponding tothe word matching the keyword from the recording medium 70 (step S165 (areading step)).

After step S165, the display unit 90 displays the text information readby the reading unit 80 such that the first part can be distinguishedfrom the second part other than the first part. Also, the sound outputunit 100 outputs a sound based on the sound information read by thereading unit 80 (step S170 (a display step and a sound output step)).

When the information recording system 10 e does not include the soundoutput unit 100, the recording unit 60 records the text information andthe conversion candidate word on the recording medium 70 such that thetext information and the conversion candidate word are associated witheach other in step S160. In step S165, the reading unit 80 reads textinformation including the first part corresponding to the word matchingthe keyword from the recording medium 70. In step S170, the display unit90 displays the text information read by the reading unit 80 such thatthe first part and the second part other than the first part can bedistinguished.

In terms of points other than the above, the processing shown in FIG. 20is similar to the processing shown in FIG. 2.

An example of a search by a search unit 65 will be described withreference to FIG. 21. In FIG. 21, a graph of a sound signal A20 isshown. In the graph, the horizontal direction represents time and thevertical direction represents amplitude. During a period T30, a periodT31, and a period T32, the amplitude of the sound signal exceeds thethreshold value. During these periods, the user utters sounds.

From the text information associated with the time point information ofthe period T30, conversion candidate words are generated by a soundprocessing unit 50. In the example shown in FIG. 21, three conversioncandidate words “experiment”, “case”, and “examination” are associatedwith the word “ten” within the text information which is a determinedword. Likewise, three conversion candidate words “securities”,“condition”, and “experiment” are associated with the word“requirements” within the text information. Likewise, a conversioncandidate word “return” is associated with the word “change” within thetext information. The word “change” is pronounced as “henkou” inJapanese. The word “return” is pronounced as “hennou” in Japanese.

Likewise, conversion candidate words are generated by the soundprocessing unit 50 from the text information associated with the timepoint information of each of the period T30 and the period T31. In FIG.21, these conversion candidate words are omitted.

When the keyword “experiment condition” is specified, the search unit 65searches for the keyword in the text information and the conversioncandidate word. The search unit 65 extracts a conversion candidate wordmatching the word “experiment” in the keyword and a conversion candidateword matching the word “condition” in the keyword from the conversioncandidate words corresponding to the period T30. Also, the search unit65 extracts words matching the term “experiment condition” within thekeyword from the text information corresponding to the period T32.

FIG. 22 shows text information displayed by the display unit 90. Thedisplay unit 90 displays text information TX10 corresponding to theperiod T30, text information TX11 corresponding to the period T31, andtext information TX12 corresponding to the period T32. The display unit90 displays a frame F10 in the first part corresponding to the wordmatching the keyword in the text information TX10. The first part in thetext information TX10 includes a determined word corresponding to aconversion candidate word matching the keyword. Also, the display unit90 displays a frame F11 in the first part corresponding to the wordmatching the keyword in the text information TX12. The first part in thetext information TX12 includes a determined word matching the keyword.In the text information TX10, a part other than the first part where theframe F10 is displayed constitutes a second part that does not includethe word corresponding to the word matching the keyword. The textinformation TX11 constitutes the second part. In the text informationTX12, a part other than the first part in which the frame F11 isdisplayed constitutes the second part.

The display unit 90 may display the first part in a first display formand the second part in a second display form different from the firstdisplay form. For example, the display unit 90 displays the first partsuch that the first part is emphasized more than the second part. Forexample, the display unit 90 displays the word within the first part ina color different from that of the word within the second part. Thedisplay unit 90 may display the word within the first part withcharacters thicker than those of the word within the second part. Thedisplay unit 90 may display the word within the first part withcharacters larger than those of the word within the second part. Amethod of distinguishing the first part from the second part is notlimited to the above-described method.

The information recording system 10 e can reduce omissions in a wordsearch in text information. As a result, the information recordingsystem 10 e can provide the user with text information corresponding tothe keyword received by a keyword reception unit 55.

As described above, the first part corresponding to the word matchingthe keyword and the second part other than the first part in the textinformation are distinguished and displayed. Thus, the user can easilycheck the search result.

The matters disclosed in the second modification of the first embodimentmay be similarly applied to the information recording system 10 c of thethird embodiment.

Fourth Embodiment

FIG. 23 shows a configuration of an information recording system 10 faccording to a fourth embodiment of the present invention. In terms ofthe configuration shown in FIG. 23, differences from the configurationshown in FIG. 15 will be described.

The information recording system 10 f shown in FIG. 23 does not includethe object information acquisition unit 20 and the image acquisitionunit 30 in the information recording system 10 d shown in FIG. 15.

In terms of points other than the above, the configuration shown in FIG.23 is similar to the configuration shown in FIG. 15.

A configuration of an information recording device 120 f is similar tothat of the 25 information recording device 120 shown in FIG. 17 exceptfor the following points. Sound information from a sound acquisitionunit 40 is input to an input unit 130. A recording unit 60 records soundinformation, text information, conversion candidate word, and time pointinformation on a recording medium 70 such that the sound information,the text information, the conversion candidate word, and the time pointinformation are associated with each other. A reading unit 80 reads textinformation including a part corresponding to a word matching thekeyword from the recording medium 70. Also, the reading unit 80 readsthe sound information associated with the time point informationcorresponding to the word matching the keyword from the recording medium70. An output unit 140 outputs the sound information and the textinformation read by the reading unit 80. That is, the output unit 140outputs the text information to a display unit 90 and outputs the soundinformation to a sound output unit 100.

The information recording system 10 f may not include the sound outputunit 100. In this case, the recording unit 60 records the textinformation and the conversion candidate word on the recording medium 70such that the text information and the conversion candidate word areassociated with each other. The reading unit 80 reads the textinformation including a part corresponding to a word matching thekeyword from the recording medium 70. The output unit 140 outputs thetext information read by the reading unit 80. Because the conversioncandidate word is associated with a determined word constituting thetext information, no time point information may be recorded.

FIG. 24 shows a procedure of processing of the information recordingdevice 120 f. In terms of the processing shown in FIG. 24, differencesfrom the processing shown in FIG. 18 will be described.

The processing in steps S200 and S205 shown in FIG. 18 is not performed.After step S210, the recording unit 60 records sound information, textinformation, conversion candidate word, and time point informationstored in a buffer within the recording unit 60 on the recording medium70 such that the sound information, the text information, the conversioncandidate word, and the time point information are associated with eachother (step S215 (a recording step)).

After step S225, the reading unit 80 reads the text informationincluding a first part corresponding to a word matching a keyword fromthe recording medium 70. Also, the reading unit 80 reads the soundinformation associated with the time point information corresponding tothe word matching the keyword from the recording medium 70 (step S245).

After step S245, the output unit 140 outputs the sound information andthe text information read by the reading unit 80. The display unit 90displays the text information output by the output unit 140 such thatthe first part can be distinguished from a second part other than thefirst part. Also, the sound output unit 100 outputs a sound based on thesound information read by the reading unit 80 (step S250 (an outputstep, a display step, and a sound output step)).

When the information recording system 10 f does not include the soundoutput unit 100, the recording unit 60 records the text information andthe conversion candidate word on the recording medium 70 such that thetext information and the conversion candidate word are associated witheach other in step S240. In step S245, the reading unit 80 reads thetext information including the first part corresponding to the wordmatching the keyword from the recording medium 70. In step S250, theoutput unit 140 outputs the text information read by the reading unit80. Also, in step S250, the display unit 90 displays the textinformation read by the reading unit 80 such that the first part and thesecond part other than the first part can be distinguished.

In terms of points other than the above, the processing shown in FIG. 24is similar to the processing shown in FIG. 18.

The information recording device 120 f can reduce omissions in a wordsearch in text information. As a result, the information recordingdevice 120 f can provide the user with text information corresponding tothe keyword received by a keyword reception unit 55.

As described above, in the text information, the first partcorresponding to the word matching the keyword and the second part otherthan the first are distinguished and displayed. Thus, the user caneasily check a search result.

The matters disclosed in the second modified example of the firstembodiment may be similarly applied to the information recording system10 f of the fourth embodiment.

(Supplement)

According to an aspect of the present invention, an informationrecording device includes an input unit, a sound processing unit, arecording unit, a keyword reception unit, a search unit, and a readingunit. Sound information based on a sound uttered by a user is input tothe input unit. The sound processing unit converts the sound informationinput to the input unit into text information and generates at least oneconversion candidate word corresponding to the sound in a process ofconverting the sound information into the text information. Therecording unit records the text information and the conversion candidateword on a recording medium such that the text information and theconversion candidate word are associated with each other. The keywordreception unit receives a keyword. The search unit searches for thekeyword received by the keyword reception unit in the text informationand the conversion candidate word and extracts a word matching thekeyword from words within the text information and the conversioncandidate word. The reading unit reads the text information including apart corresponding to the word matching the keyword from the recordingmedium.

According to an aspect of the present invention, an informationrecording device includes an input unit, a sound processing unit, arecording unit, a keyword reception unit, a search unit, and a readingunit. Object information about an object and sound information based ona sound uttered by an observer who observes the object are input to theinput unit. The sound processing unit converts the sound informationinput to the input unit into text information and generates at least oneconversion candidate word corresponding to the sound in a process ofconverting the sound information into the text information. Therecording unit records the object information, the text information, theconversion candidate word, and time point information on a recordingmedium such that the object information, the text information, theconversion candidate word, and the time point information are associatedwith each other. The time point information indicates a time point atwhich the object information was acquired and a time point at which thesound information based on the text information was acquired. Thekeyword reception unit receives a keyword. The search unit searches forthe keyword received by the keyword reception unit in the textinformation and the conversion candidate word and extracts a wordmatching the keyword from words within the text information and theconversion candidate word. The reading unit reads the object informationassociated with the time point information corresponding to the wordmatching the keyword from the recording medium.

According to an aspect of the present invention, an informationrecording method includes an object information acquisition step, asound acquisition step, a sound processing step, a recording step, akeyword reception step, a search step, a reading step, and a displaystep. In the object information acquisition step, an object informationacquisition unit acquires object information about an object. In thesound acquisition step, a sound acquisition unit acquires soundinformation based on a sound uttered by an observer who observes theobject. In the sound processing step, a sound processing unit convertsthe sound information acquired by the sound acquisition unit into textinformation and generates at least one conversion candidate wordcorresponding to the sound in a process of converting the soundinformation into the text information. In the recording step, arecording unit records the object information, the text information, theconversion candidate word, and time point information on a recordingmedium such that the object information, the text information, theconversion candidate word, and the time point information are associatedwith each other. The time point information indicates a time point atwhich the object information was acquired and a time point at which thesound information that is a source of the text information was acquired.In the keyword reception step, a keyword reception unit receives akeyword. In the search step, a search unit searches for the keywordreceived by the keyword reception unit in the text information and theconversion candidate word and extracts a word matching the keyword fromwords within the text information and the conversion candidate word. Inthe reading step, a reading unit reads the object information associatedwith the time point information corresponding to the word matching thekeyword from the recording medium. In the display step, a display unitdisplays the text information read by the reading unit.

According to an aspect of the present invention, an informationrecording method includes an input step, a sound processing step, arecording step, a keyword reception step, a search step, and a readingstep. In the input step, sound information based on a sound uttered by auser is input to an input unit. In the sound processing step, a soundprocessing unit converts the sound information input to the input unitinto text information and generates at least one conversion candidateword corresponding to the sound in a process of converting the soundinformation into the text information. In the recording step, arecording unit records the text information and the conversion candidateword on a recording medium such that the text information and theconversion candidate word are associated with each other. In the keywordreception step, a keyword reception unit receives a keyword. In thesearch step, a search unit searches for the keyword received by thekeyword reception unit in the text information and the conversioncandidate word and extracts a word matching the keyword from wordswithin the text information and the conversion candidate word. In thereading step, a reading unit reads the text information including a partcorresponding to the word matching the keyword from the recordingmedium.

According to an aspect of the present invention, an informationrecording method includes an input step, a sound processing step, arecording step, a keyword reception step, a search step, and a readingstep. In the input step, object information about an object and soundinformation based on a sound uttered by an observer who observes theobject are input to an input unit. In the sound processing step, a soundprocessing unit converts the sound information input to the input unitinto text information and generates at least one conversion candidateword corresponding to the sound in a process of converting the soundinformation into the text information. In the recording step, arecording unit records the object information, the text information, theconversion candidate word, and time point information on a recordingmedium such that the object information, the text information, theconversion candidate word, and the time point information are associatedwith each other. The time point information indicates a time point atwhich the object information was acquired and a time point at which thesound information that is a source of the text information was acquired.In the keyword reception step, a keyword reception unit receives akeyword. In the search step, a search unit searches for the keywordreceived by the keyword reception unit in the text information and theconversion candidate word and extracts a word matching the keyword fromwords within the text information and the conversion candidate word. Inthe reading step, a reading unit reads the object information associatedwith the time point information corresponding to the word matching thekeyword from the recording medium.

While preferred embodiments of the invention have been described andshown above, it should be understood that these are exemplars of theinvention and are not to be considered as limiting. Additions,omissions, substitutions, and other modifications can be made withoutdeparting from the spirit or scope of the present invention.Accordingly, the invention is not to be considered as being limited bythe foregoing description, and is only limited by the scope of theappended claims.

What is claimed is:
 1. An information recording system comprising: asound acquisition unit configured to acquire sound information based ona sound uttered by a user; a sound processing unit configured to convertthe sound information acquired by the sound acquisition unit into textinformation and generate at least one conversion candidate wordcorresponding to the sound in a process of converting the soundinformation into the text information; a recording unit configured torecord the text information and the conversion candidate word such thatthe text information and the conversion candidate word are associatedwith each other; a keyword reception unit configured to receive akeyword; a search unit configured to search for the keyword received bythe keyword reception unit in the text information and the conversioncandidate word and extract a word matching the keyword from words withinthe text information and the conversion candidate word; a reading unitconfigured to read the text information including a first partcorresponding to the word matching the keyword and a second part otherthan the first part from the recording medium; and a display unitconfigured to display the text information read by the reading unit suchthat the first part and the second part are able to be distinguished. 2.An information recording system comprising: an object informationacquisition unit configured to acquire object information about anobject; a sound acquisition unit configured to acquire sound informationbased on a sound uttered by an observer who observes the object; a soundprocessing unit configured to convert the sound information acquired bythe sound acquisition unit into text information and generate at leastone conversion candidate word corresponding to the sound in a process ofconverting the sound information into the text information; a recordingunit configured to record the object information, the text information,the conversion candidate word, and time point information on a recordingmedium such that the object information, the text information, theconversion candidate word, and the time point information are associatedwith each other, the time point information indicating a time point atwhich the object information was acquired and a time point at which thesound information that is a source of the text information was acquired;a keyword reception unit configured to receive a keyword; a search unitconfigured to search for the keyword received by the keyword receptionunit in the text information and the conversion candidate word andextract a word matching the keyword from words within the textinformation and the conversion candidate word; a reading unit configuredto read the object information associated with the time pointinformation corresponding to the word matching the keyword from therecording medium; and a display unit configured to display the textinformation read by the reading unit.
 3. The information recordingsystem according to claim 2, wherein the recording unit records theobject information, the sound information, the text information, theconversion candidate word, and the time point information on therecording medium such that the object information, the soundinformation, the text information, the conversion candidate word, andthe time point information are associated with each other, the timepoint information indicating a time point at which the objectinformation was acquired, a time point at which the sound informationwas acquired, and a time point at which the sound information that is asource of the text information was acquired, the information recordingsystem further comprises an event detection unit configured to detect atleast two events on the basis of at least one piece of the objectinformation and the sound information recorded on the recording medium,the events being a state in which the at least one piece of the objectinformation and the sound information recorded on the recording mediumsatisfies a predetermined condition, the reading unit reads at least twopieces of the object information, each of which is associated with thetime point information corresponding to an event occurrence time pointthat is a time point at which the event occurred from the recordingmedium, the display unit displays the object information read by thereading unit, the search unit searches for the keyword received by thekeyword reception unit in the text information and the conversioncandidate word that are associated with the time point informationcorresponding to the event occurrence time point, and the display unitdisplays the object information associated with the time pointinformation corresponding to the event occurrence time point at whichthe word matching the keyword was detected such that the objectinformation is distinguished from other object information.
 4. Theinformation recording system according to claim 2, wherein the recordingunit records the object information, the sound information, the textinformation, the conversion candidate word, and the time pointinformation on the recording medium such that the object information,the sound information, the text information, the conversion candidateword, and the time point information are associated with each other, thetime point information indicating a time point at which the objectinformation was acquired, a time point at which the sound informationwas acquired, and a time point at which the sound information that is asource of the text information was acquired, the information recordingsystem further comprises an event detection unit configured to detect atleast two events on the basis of at least one piece of the objectinformation and the sound information recorded on the recording medium,the events being a state in which the at least one piece of the objectinformation and the sound information recorded on the recording mediumsatisfies a predetermined condition, the reading unit reads the objectinformation and the text information that are associated with the timepoint information corresponding to an event occurrence time point thatis a time point at which the event occurred from the recording medium,the display unit displays the object information and the textinformation read by the reading unit, the search unit searches for thekeyword received by the keyword reception unit in the text informationand the conversion candidate word that are associated with the timepoint information corresponding to the event occurrence time point, andthe display unit displays the word that is within the text informationand matches the keyword such that the word is distinguished from otherwords within the text information, and displays the word that is withinthe text information and is associated with the conversion candidateword matching the keyword such that the word is distinguished from otherwords within the text information.
 5. The information recording systemaccording to claim 3, wherein, when a state of the object indicated bythe object information is a state predefined as an event detectioncondition, the event detection unit detects the event.
 6. Theinformation recording system according to claim 4, wherein, when a stateof the object indicated by the object information is a state predefinedas an event detection condition, the event detection unit detects theevent.
 7. The information recording system according to claim 3, whereinthe sound information is a time-series sound signal, and when amplitudeor power of the sound signal exceeds a threshold value predefined as anevent detection condition, the event detection unit detects the event.8. The information recording system according to claim 4, wherein thesound information is a time-series sound signal, and when amplitude orpower of the sound signal exceeds a threshold value predefined as anevent detection condition, the event detection unit detects the event.9. The information recording system according to claim 3, wherein, whena sound indicated by the sound information matches a sound of a keywordpredefined as an event detection condition, the event detection unitdetects the event.
 10. The information recording system according toclaim 4, wherein, when a sound indicated by the sound informationmatches a sound of a keyword predefined as an event detection condition,the event detection unit detects the event.
 11. The informationrecording system according to claim 2, wherein the reading unit readsthe object information and the text information that are associated withthe time point information corresponding to the word matching thekeyword from the recording medium, the display unit displays the objectinformation and the text information read by the reading unit, and thedisplay unit displays the word that is within the text information andmatches the keyword such that the word is distinguished from other wordswithin the text information, and displays the word that is within thetext information and is associated with the conversion candidate wordmatching the keyword such that the word is distinguished from otherwords within the text information.
 12. The information recording systemaccording to claim 2, wherein the recording unit records the objectinformation, the sound information, and the time point information onthe recording medium such that the object information, the soundinformation, and the time point information are associated with eachother, the time point information indicating a time point at which theobject information was acquired and a time point at which the soundinformation was acquired, the reading unit reads the sound informationfrom the recording medium, the sound processing unit converts the soundinformation read by the reading unit into the text information andgenerates the conversion candidate word, and the recording unitassociates the text information and the conversion candidate word withthe object information and the time point information recorded on therecording medium and records the text information and the conversioncandidate word on the recording medium, the time point information withwhich the text information is associated indicates a time point at whichthe sound information that is a source of the text information wasacquired.
 13. An information recording method comprising: a soundacquisition step in which a sound acquisition unit acquires soundinformation based on a sound uttered by a user; a sound processing stepin which a sound processing unit converts the sound information acquiredby the sound acquisition unit into text information and generates atleast one conversion candidate word corresponding to the sound in aprocess of converting the sound information into the text information; arecording step in which a recording unit records the text informationand the conversion candidate word on a recording medium such that thetext information and the conversion candidate word are associated witheach other; a keyword reception step in which a keyword reception unitreceives a keyword; a search step in which a search unit searches forthe keyword received by the keyword reception unit in the textinformation and the conversion candidate word and extracts a wordmatching the keyword from words within the text information and theconversion candidate word; a reading step in which a reading unit readsthe text information including a first part corresponding to the wordmatching the keyword and a second part other than the first part fromthe recording medium; and a display step in which a display unitdisplays the text information read by the reading unit such that thefirst part and the second part are able to be distinguished.